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ABSTRACT 

Knowledge  Management  (KM)  has  been  identified  as  one  of  several  enablers  of 
the  current  Transformation  effort  in  the  Department  of  Defense.  Several  organizations 
within  the  DoD  have  started  using  KM  and  are  now  interested  in  identifying  and, 
subsequently,  measuring  the  benefits  in  order  to  gauge  success.  While  many  studies  have 
been  undertaken  to  identify  the  benefits  of  KM  in  the  commercial  sector,  similar  efforts 
to  investigate  the  benefits  in  a  DoD  context  are  lacking.  Using  a  Delphi  study  involving 
key  DoD  KM  experts,  this  research  aims  to  identify  the  major  realized  benefits  associated 
with  KM  practice  from  a  strictly  DoD  perspective. 
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IDENTIFYING  THE  BENEFITS  OF  KNOWLEDGE  MANAGEMENT  IN  THE 


DEPARTMENT  OF  DEFENSE:  A  DELPHI  STUDY 


I.  INTRODUCTION 


Background 

The  U.S.  Department  of  Defense  (DoD)  is  eurrently  undergoing  a  transformation 
which  aims  to  revolutionize  the  way  it  fights  wars  and  conducts  business.  This 
transformation  is  “the  process  whereby  the  DoD  is  overhauling  the  U.S.  military  and 
defense  establishment  to  enable  it  to  counter  21st  century  threats  most  effectively. 
Transformation  is  about  new  ways  of  thinking,  fighting,  and  organizing  the  Department 
and  its  operations  -  as  well  as  about  acquiring  new  system  capabilities”  (Defense  Link, 
2003).  Several  organizations  across  the  DoD  are  already  beginning  to  identify  the 
application  of  knowledge  management  (KM)  as  a  key  strategic  focus  in  order  to  affect 
this  transformation  (Bartczak,  2002;  Cuviello  &  Michaliga,  2003). 

Knowledge  management  is  “the  attempt  to  recognize  what  is  essentially  a  human 
asset  buried  in  the  minds  of  individuals,  and  leverage  it  into  an  organizational  asset  that 
can  be  accessed  and  used  by  a  broader  set  of  individuals  on  whose  decisions  the  firm 
depends”  (Davenport  &  Prusak,  1998).  The  ultimate  goal  of  KM  is  to  take  advantage  of 
this  knowledge  asset  in  order  to  provide  some  level  of  benefit  to  the  organization 
(Davenport  &  Prusak,  1998;  Nonaka,  1996).  However,  many  in  both  the  military  and 
civilian  information  resource  management  communities  believe  that  KM  provides 
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nothing  beyond  what  is  already  aeeomplished  with  information  management,  and  is 
simply  “old  wine  in  a  new  bottle”  (MeAdam  &  MeCreedy,  1999;  Spiegler,  2000). 
Consequently,  leaders  and  managers  would  like  proof  that  KM  works,  and  identifying 
and  measuring  the  benefits  of  KM  is  recognized  as  a  key  issue  for  future  KM  research 
(Edwards,  Handzic,  Carlsson,  &  Nissen,  2003;  Firestone  &  McElroy,  2003). 

Given  that  serious  resources  are  already  being  committed  to  KM  in  the  DoD,  it 
follows  that  identifying  the  benefits  derived  from  KM  from  a  DoD  perspective  is  highly 
desirable.  Indeed,  Bartczak  (2002)  as  well  as  Bennet  and  Porter  (2003)  found  that 
demonstrating  a  return  on  investment  regarding  KM  initiatives  in  the  DoD  is  necessary  in 
order  to  gain  and  keep  leadership  support  as  well  as  garner  funding  commitments. 
Certainly  this  would  require  an  ability  to  measure  the  benefit  such  initiatives  provide. 
However,  this  task  cannot  be  addressed  until  an  initial  identification  of  such  benefits  has 
been  accomplished.  While  a  preliminary  literature  review  identified  several  studies 
which  attempted  to  quantify  the  benefits  of  KM  from  a  commercial  perspective,  no 
similar  research  taking  a  strictly  DoD  perspective  was  identified. 

Research  Questions 

1)  What  does  the  literature  identify  as  the  key  benefits  of  KM  programs  in  general? 

2)  What  do  DoD  KM  experts  identify  as  the  key  benefits  associated  with  KM  in  the 
DoD? 

3)  Do  DoD  KM  experts  experience  problems  measuring  KM  benefits?  If  so,  how? 
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Research  Approach 

A  Delphi  study  will  be  condueted  consisting  of  two  questionnaires  distributed  to 
several  knowledge  management  experts  across  the  DoD.  Experts  will  be  identified  as 
those  currently  overseeing  knowledge  management  initiatives  in  their  respective 
organizations,  and  having  at  least  two  years  of  experience  working  with  knowledge 
management  in  general.  The  Delphi  study  will  be  used  to  identify  the  degree  to  which 
DoD  KM  experts  agree  that  KM  benefits  identified  in  commercial  organizations  apply  as 
benefits  to  the  DoD. 

Benefits/Implications  of  Research 

Given  the  amount  of  time  and  money  the  DoD  is  investing  in  knowledge 
management  initiatives,  a  method  of  quantifying  and  measuring  the  benefits  that  result 
from  these  investments  is  highly  desirable.  Academics  have  identified  a  strong  need  for 
research  regarding  this  subject,  and  a  preliminary  literature  review  did  indeed  uncover 
attempts  to  identify  KM  benefits  from  a  commercial  perspective.  However,  nothing 
identifying  the  applicability  of  these  benefits  to  the  DoD  was  found.  The  results  of  this 
research  could  be  used  as  a  first  step  to  thoroughly  identify  KM  benefits  to  the  DoD,  and 
ultimately  a  process  for  measuring  and  calculating  a  return  on  investment  of  DoD  KM 


initiatives. 


Thesis  Overview 


The  remainder  of  this  document  will  report  the  efforts  to  address  the  research 
questions  presented  in  this  chapter.  In  chapter  II,  the  literature  from  scholars  that  serves 
as  the  theoretical  foundations  of  this  work  will  be  reviewed.  Specifically,  a  general 
review  of  KM  will  be  provided,  followed  by  a  review  of  KM  applications  in  the  DoD,  the 
benefits  of  KM  identified  thus  far,  and  a  discussion  regarding  the  difficulties  of 
measuring  KM  performance.  Chapter  III  presents  the  research  methodology  used  in  this 
study,  while  chapter  IV  sets  forth  a  detailed  analysis  of  the  collected  data  and  the  findings 
that  resulted  from  this  analysis.  Finally,  the  thesis  closes  with  conclusions  and 
recommendations. 
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II.  LITERATURE  REVIEW 


The  Knowledge  Society 

Peter  Drucker  (1993)  argues  that  our  world  as  we  know  it  has  very  reeently 
undergone  a  shift  from  an  information  to  a  knowledge  soeiety.  His  idea  is  that 
knowledge  has  become  the  resource  driving  our  economy.  The  ability  to  apply 
knowledge  in  order  to  improve  processes,  or  even  generate  new  knowledge,  is  what 
determines  a  successful  company  today.  Just  as  the  laborer  was  the  key  enabler  during 
the  industrial  revolution,  the  “knowledge  worker”  has  become  the  key  enabler  in  this  new 
knowledge  society. 

Several  other  leading  business  authors  share  the  views  of  Drucker.  Toffler  (1990) 
sees  the  primary  source  of  power  in  today’s  world  shifting  from  might  and  wealth 
towards  knowledge.  This  shift  in  the  nature  of  power  will  drive  countries  and  companies 
to  scramble  for  control  of  new  knowledge  resources.  Reich  (1991)  argues  that  today’s 
economy  is  global  and  knows  no  borders.  Thus,  competitive  advantage  can  only  be 
gained  by  those  who  can  use  knowledge  to  identify,  solve,  and  broker  new  problems. 
Quinn  (1992)  states  simply  that,  “Intellect  is  the  core  resource  in  producing  and 
delivering  service.”  Finally,  Thomas  Stewart  believes  that,  “knowledge  has  become  the 
preeminent  economic  resource  -  more  important  than  raw  material;  more  important, 
often,  than  money”  (Stewart,  1997). 


Knowledge 

If  we  are  to  believe  that  the  foeus  of  sueeess  in  today’s  world  is  squarely  upon 
gaining  and  leveraging  knowledge,  then  it  is  probably  in  our  best  interest  to  attaeh  some 
meaning  to  the  term.  This  is  quite  a  bit  more  diffleult  than  it  sounds,  however,  as  a  great 
many  definitions  exist  throughout  the  body  of  literature.  Some  examples  found  during 
this  literature  review  inelude: 

■  Knowledge  is  aetionable  information  that  is  possessed  in  the  mind  (Nonaka  & 
Takeuehi,  1995). 

■  . .  .knowledge  is  information  effeetive  in  action,  information  focused  on  results 
(Drucker,  1993). 

■  Knowledge  is  the  experience,  concepts,  values,  or  beliefs  that  increases  an 
individual’s  capability  to  take  effective  action  (Alavi  &  Leidner,  1999). 

■  Knowledge  is  a  fluid  mix  of  framed  experience,  values,  contextual  information, 
and  expert  insight  that  provides  a  framework  for  evaluating  and  incorporating 
new  experiences  and  information.  It  originates  and  is  applied  in  the  minds  of 
knowers.  In  organizations,  it  often  becomes  embedded  not  only  in  documents  or 
repositories  but  also  in  organizational  routines,  processes,  practices,  and  norms 
(Davenport  &  Prusak,  1998). 

■  Knowledge  is  a  dynamic  human  process  of  justifying  personal  belief  toward  the 


“truth”  (Nonaka,  1996). 


■  Knowledge  is  the  proeess  of  knowing,  a  reflexive  process  that  takes  data  and 
information,  in  a  social  context  . . .  and  generates  new  data,  information,  and/or 
knowledge  (Spiegler,  2000). 

Although  it  is  difficult  to  come  up  with  a  concise  statement  regarding  knowledge, 
several  common  themes  can  certainly  be  identified  throughout  the  literature  when 
referring  to  organizational  knowledge: 

1.  Knowledge  creation  is  a  decidedly  human-centric  process.  In  a  strict  sense, 
knowledge  can  be  created  only  by  individuals  (Davenport  &  Prusak,  1998; 
Nonaka  &  Takeuchi,  1995). 

2.  Knowledge  is  inextricably  linked  to  both  data  and  information.  While  the 
majority  of  knowledge  management  scholars  see  knowledge  as  the  final  stage  in 
an  evolution  that  turns  data  into  information  into  knowledge  (Davenport  & 
Prusak,  1998;  Nonaka  &  Takeuchi,  1995),  Tuomi  (2000)  actually  prefers  a 
reverse  hierarchy,  posing  that  knowledge  must  first  exist  before  information  and 
data  can  be  generated. 

3.  If  something  is  to  be  considered  knowledge,  it  must  have  the  capacity  to  be  acted 
upon  (Alavi  &  Leidner,  1999;  Drucker,  1993). 

Further  complicating  matters  is  the  idea  that  knowledge  can  be  categorized  into 
two  separate  types.  The  first  is  tacit  knowledge,  which  is  personal  in  nature  and 
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generally  hard  to  formalize  and  eommunicate.  It  exists  most  readily  in  the  human  mind 
(Polanyi,  1966).  An  example  of  tacit  knowledge  would  be  a  baseball  pitcher 
understanding  exactly  how  to  throw  a  curveball  for  a  strike  on  a  warm  sunny  day  as 
opposed  to  a  cold  rainy  day.  While  the  basic  technique  for  throwing  the  ball  can  indeed 
be  imparted  upon  another,  the  distinct  difference  between  what  will  result  in  a  strike  on  a 
sunny  as  opposed  to  rainy  day  is  almost  impossible  to  explain.  It  is  something  that 
cannot  easily  be  transferred  or  communicated  through  written  or  oral  means,  and  is  most 
likely  understood  only  through  repeated  experience. 

The  second  category  is  explicit  knowledge,  which  can  also  be  referred  to  as 
“codified”  knowledge.  Knowledge  becomes  codified  when  it  has  been  recorded 
somehow,  whether  written  down  on  paper  or  embodied  in  an  artifact  of  some  sort. 
Explicit  knowledge  is  easily  communicated  through  formalized  language  (Polanyi,  1966). 
When  an  individual  understands  a  particular  bit  of  tacit  knowledge  well  enough  to 
articulate  it  to  another,  explicit  knowledge  is  being  transferred. 

Knowledge  Management 

If  today’s  organizations  should  focus  on  gaining  and  leveraging  knowledge,  it 
follows  that  there  is  an  interest  in  managing  its  generation  as  well  as  preservation  within 
the  organization.  This  leads  to  the  next  step  of  this  literature  review,  which  is  directed 
toward  the  practice  of  knowledge  management.  Finding  a  single  definition  for  the  term 
proved  to  be  as  elusive  as  searching  for  a  definition  of  knowledge.  The  following  are 
examples  of  knowledge  management  definitions  from  some  top  researchers  in  the  field: 
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■  Management  of  organizational  knowledge  for  creating  business  value  and 
generating  a  competitive  advantage  (Tiwana,  2000). 

■  Knowledge  management  can  be  viewed  as  turning  data  (raw  material)  into 
information  (finished  goods)  into  knowledge  (actionable  finished  goods)  (Kanter, 
1999). 

■  [knowledge  management  is] . .  .finding  and  growing  intellectual  capital,  storing  it, 
selling  it,  sharing  it...  (Stewart,  1997) 

■  Knowledge  management  includes  not  only  the  acquisition,  accumulation,  and 
utilization  of  existing  knowledge,  but  also  the  creation  of  new  knowledge  (Nonaka 
&  Takeuchi,  1995). 

Davenport  and  Prusak  provide  a  definition  that  ties  together  common  themes  from  the 
previous  definitions: 

“Knowledge  management  is  the  attempt  to  recognize  what  is  essentially  a 
human  asset  buried  in  the  minds  of  individuals,  and  leverage  it  into  an 
organizational  asset  that  can  be  accessed  and  used  by  a  broader  set  of 
individuals  on  whose  decisions  the  firm  depends”  (Davenport  &  Prusak, 

1998). 


Q 


Indeed,  the  goal  of  knowledge  management  is  to  share  the  knowledge  that  is  eontained 
within  the  minds  of  the  individual,  and  make  it  available  to  the  entire  organization. 
Ultimately  this  should  lead  to  better  deeisions,  whieh  ean  lead  to  greater  organizational 
sueeess. 

The  initial  diffieulty  with  managing  knowledge  is  identifying  how  the  proeess 
differs  in  any  way  from  information  management.  In  fact,  some  organizations  still  think 
of  knowledge  and  information  management  as  one  and  the  same  (Spiegler,  2000). 
Knowledge  management  is  unique,  however,  in  that  it  requires  us  to  somehow  capture, 
store,  and  disseminate  something  that  by  it’s  very  nature  resides  in  a  person’s  mind 
(Alavi  &  Leidner,  1999).  This  is  what  makes  the  practice  of  knowledge  management 
such  a  challenge,  though  several  technologies  are  available  to  assist  in  the  effort. 

Some  of  the  most  common  technologies  that  support  organizational  knowledge 
management  include  knowledge  repositories,  knowledge  maps,  and  expert  systems. 
Knowledge  repositories  are  basically  large  databases  containing  structured,  explicit 
knowledge,  usually  in  document  form.  Probably  the  best  example  of  a  knowledge 
repository,  which  is  also  most  likely  the  largest,  is  the  internet  (Davenport  &  Prusak, 
1998).  Using  the  internet,  individuals  can  access  vast  amounts  of  knowledge  contained 
in  repositories  spread  across  the  entire  globe. 

Knowledge  maps  can  be  thought  of  as  a  repository  of  another  form.  Rather  than 
containing  explicit  knowledge  itself,  however,  knowledge  maps  direct  the  user  to  a 
specific  individual  or  individuals  who  possesses  the  desired  knowledge.  In  a  sense, 
knowledge  maps  do  just  what  the  name  implies;  they  assist  a  user  in  locating  those  who 
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possesses  knowledge.  This  technology  readily  facilitates  the  transfer  of  tacit  knowledge, 
since  often  a  higher  level  of  knowledge  transfer  can  take  place  via  personal 
communication  (Nonaka  &  Takeuchi,  1995). 

Expert  systems  are  a  different  sort  of  knowledge  management  technology,  in  that 
they  aim  to  emulate  the  application  of  expert  knowledge  toward  a  particular  process.  For 
example,  McDonnell  Douglas  once  developed  an  expert  system  that  aimed  to  scan  an 
aircraft  approaching  the  runway  for  landing  in  order  to  determine  if  it  was  positioned 
properly.  This  is  something  that  experienced  ground  crews  could  do  at  a  glance. 
However,  it  took  McDonnell  Douglas  two  years  to  develop  a  system  that  was  80  to  85 
percent  as  accurate  as  a  two  second  human  glance  (Davenport  &  Prusak,  1998).  This 
highlights  a  shortcoming  of  expert  systems,  which  is  that  it’s  extremely  difficult,  if  not 
impossible,  to  develop  a  machine  that  can  replace  fundamentally  human  thinking 
processes.  Thus,  the  input  and  participation  of  organizational  members  seems  to  remain 
an  essential  component  in  the  knowledge  management  process. 

Technology-  vs.  People-Based  KM 

Recognizing  that  knowledge  management  required  more  than  just  the  latest  and 
greatest  technology  led  to  the  identification  of  two  major  KM  process  areas.  In  addition 
to  “technology -based”  processes,  which  were  discussed  above,  Sveiby  (1997)  has 
identified  the  notion  of  “people-based”  processes.  Since  a  key  component  of  knowledge 
management  involves  knowledge  generation,  which  in  turn  is  dependent  upon  human 
interaction,  it  became  clear  that  a  focus  on  appropriate  personnel  management  techniques 


was  as  important,  if  not  more  so,  than  the  technologies  themselves.  In  other  words, 

.  .effective  knowledge  management  cannot  take  place  without  extensive  behavioral, 
cultural,  and  organizational  change”  (Davenport  &  Prusak,  1998).  This  notion  has  led  to 
a  new  role  for  managers  in  an  organization.  No  longer  may  they  only  be  considered 
managers  of  personnel,  but  managers  of  knowledge  as  well.  In  fact,  according  to 
Drucker,  “supplying  knowledge  to  find  out  how  existing  knowledge  can  best  be  applied 
to  produce  results  is,  in  effect,  what  we  mean  by  management”  (Drucker,  1993).  He  goes 
on  to  state  that  the  definition  of  a  manager  is,  “one  who  is  responsible  for  the  application 
and  performance  of  knowledge.” 

Knowledge  Management  in  the  DoD 

Knowledge  Management  is  still  a  relatively  new  concept  in  today’s  Department 
of  Defense,  and  each  service  is  approaching  it  independently  (Defense,  2000).  Many 
DoD  organizations  have  managed  to  implement  several  technical  KM  solutions,  while  a 
few  are  working  toward  institutionalizing  a  more  far-reaching  enterprise-wide  knowledge 
strategy.  United  States  Joint  Forces  Command’s  (USJFCOM)  “Knowledge  Today”  is 
considered  the  roots  of  KM  for  the  DoD  (DoD/CIO,  2001).  This  system  is  essentially  a 
knowledge  repository  which  allows  the  capture,  maintenance,  and  sharing  of  knowledge 
throughout  the  command. 

Both  the  Army  and  the  Navy  have  taken  big  steps  to  not  only  implement  several 
knowledge  management  technologies,  but  also  incorporate  enterprise-level  KM 
strategies.  Army  Knowledge  Online  (AKO)  is  perhaps  the  most  visible  result  of  the 
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Army’s  strong  focus  on  KM  initiatives.  AKO  began  as  an  information-only  website,  but 
has  sinee  expanded  into  a  knowledge  portal  and  eollaboration  platform  for  the  entire 
Army  (http://www.ako.armv.mill  (Bartczak,  2002).  AKO  is  one  of  several  knowledge 
strategies  the  Army  has  implemented  on  the  road  its  transformation  into  a  knowledge- 
based  organization  (Bower,  2001). 

The  Navy  has  also  fielded  a  web  portal  known  as  Navy  Knowledge  Online 
(NKO)  (http://www.nko.navv.mil).  as  well  as  the  Navy  Marine  Corps  Portal.  Like  the 
Army,  the  Navy  has  worked  hard  to  develop  an  enterprise-wide  knowledge  strategy,  with 
a  goal  of  transforming  the  Navy  into  a  “knowledge-eentrie  organization”  (Crupi,  Hedges, 
Passen,  Thornton,  &  White,  2001). 

The  Air  Foree,  on  the  other  hand,  has  not  taken  sueh  lengthy  steps  toward 
adopting  an  enterprise  knowledge  strategy  (Bartezak,  2002).  However,  the  eurrent  Air 
Foree  Information  Strategy  (2002)  does  indeed  identify  a  goal  to  “. .  .implement 
knowledge  management  praetiees  and  teehnologies  to  assure  knowledge  is  identified, 
captured,  and  shared.”  Perhaps  the  most  tangible  evidence  of  the  Air  Force’s  adoption  of 
knowledge  management  is  the  Air  Foree ’s  version  of  a  knowledge  portal  known  as  Air 
Foree  Knowledge  Now  (AFKN)  (https://afkm.wpafb.af.mil).  Unlike  the  Army  and  Navy 
portals,  AFKN  is  only  aceessible  from  a  .mil  domain. 

While  a  focus  on  an  enterprise-wide  knowledge  strategy  may  not  be  universal,  it 
is  elear  that  at  a  minimum  several  knowledge  management  teehnologies  are  being 
implemented  aeross  the  DoD. 
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Benefits  of  Knowledge  Management 

It  would  make  no  sense  to  implement  KM  if  there  were  no  assoeiated  benefits. 
Initial  KM  researeh,  however,  identifies  numerous  positive  outeomes  for  organizations 
praetieing  knowledge  management.  Initial  reports  on  the  benefits  of  KM  were  largely  in 
the  form  of  case  studies,  with  perhaps  the  most  widely  reported  benefits  being  an  increase 
in  productivity  as  well  as  a  decrease  in  costs  (Davenport  &  Prusak,  1998;  Nonaka  & 
Takeuchi,  1995).  For  example,  Hoffmann-LaRoche,  a  Swiss  pharmaceutical  firm,  credits 
a  knowledge  management  initiative  in  1993-1994  with  saving  them  $1  million  per  day. 
The  firm  implemented  a  system  which  reduced  the  application  time  for  new  FDA  drug 
approvals  by  several  months  (Davenport  &  Prusak,  1998).  Similarly,  Hewlett-Packard 
used  a  KM  system  to  streamline  their  customer  support  process,  reducing  call  times  by 
two-thirds  and  cutting  the  cost  per  call  by  50  percent  (Davenport  &  Prusak,  1998). 

Another  strongly  reported  benefit  of  KM  is  the  acceleration  of  internal  processes. 
Knowledge  shared  both  within  and  between  organizations  ultimately  results  in  the 
creation  of  new  knowledge  to  help  drive  the  organization  forward  (Nonaka  &  Takeuchi, 
1995).  Matsushita  was  able  to  develop  the  first  automatic  bread  maker  at  an  accelerated 
pace  because  of  a  KM  process  which  enabled  employees  to  make  first  hand  observations 
of  a  master  bread-maker  at  work  (Nonaka  &  Takeuchi,  1995). 

More  recently,  as  the  KM  field  has  grown  and  matured,  a  number  of  quantitative 
studies  emerged  attempting  to  define  a  list  of  benefits  related  to  KM.  In  a  1999  report, 
KPMG  Consulting  surveyed  423  organizations  across  the  UK,  mainland  Europe,  and  the 
US  regarding  several  knowledge  management  issues,  including  benefits  achieved  through 
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KM  (KPMG  Consulting,  1999).  After  identifying  each  organization’s  expected  benefits 
through  the  use  of  KM,  the  respondents  reported  the  actual  realized  benefits.  The  top 
four  benefits  included  better  decision  making,  faster  response  to  key  business  issues, 
better  customer  handling,  and  improved  employee  skills.  Figure  2.1  provides  a  snapshot 
of  the  expected  benefits,  while  Figure  2.2  outlines  the  entire  list  of  realized  benefits. 
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Figure  2.1  -  Expected  KM  benefits  (KPMG  Consulting,  1999) 
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Figure  2.2  -  Realized  benefits  of  KM  (KPMG  Consulting,  1999) 


A  reeent  study  by  North  and  Homung  (2003)  investigated  KM  benefits  reported 
by  34  German  eompanies  who  qualified  to  eompete  for  the  German  Commerzbank 
“Knowledge  Manager  2002”  award.  Partieipating  companies  were  divided  into  three 
categories:  small  firms  with  less  than  50  employees,  medium-sized  firms  with  no  more 
than  250  employees,  and  large  firms  with  more  than  250  employees.  A  primary  focus  of 
the  application  was  identifying  KM  benefits  for  employees,  customers,  and  the 
organization  itself  Benefits  were  based  upon  a  balanced  scorecard  approach  with  four 
categories:  business  processes,  employee  satisfaction,  customer  satisfaction,  and  financial 
results.  Figures  2.3  -  2.6  outline  the  number  of  firms  reporting  the  respective  benefits. 


Figure  2.3  -  Number  of  firms  reporting  a  benefit  in  the  area  of  business  proeesses  (North 

&  Homung,  2003) 


Figure  2.4  -  Number  of  firms  reporting  a  benefit  in  the  area  of  employee  satisfaetion 

(North  &  Homung,  2003) 
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Figure  2.5  -  Number  of  firms  reporting  a  benefit  in  the  area  of  customer  satisfaction 

(North  &  Homung,  2003) 


Figure  2.6  -  Number  of  firms  reporting  a  benefit  in  the  area  of  financial  results  (North  & 

Homung,  2003) 


North  and  Homung ’s  study  found  that  the  benefits  knowledge  management 
provided  to  an  organization  depended  upon  the  KM  approach  taken  (Figure  2.7).  The 
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three  approaehes  identified  were  either  IT-eentered,  KM  applied  to  specifie  problem 
areas,  or  professional  KM.  Professional  KM  was  defined  as  balaneing  the  necessary  IT 
with  proper  management  incentives  so  that  KM  is  integrated  into  all  business  process  and 
projects  across  the  organizations  (North  &  Homung,  2003).  Indeed,  this  approach 
resulted  in  the  most  reported  benefits  as  well  as  the  most  balanced  realization  of  benefits 
across  five  categories. 
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Figure  2.7  -  Benefits  depending  upon  the  KM  approach  taken  (North  &  Homung,  2003) 


The  balanced  scorecard  approach  used  by  North  and  Homng  to  categorize  KM 
benefits  is  also  supported  by  a  report  from  Lopez  and  Rayboum  (2003)  of  the  American 
Productivity  and  Quality  Center  (APQC).  Under  the  APQC  KM  balanced  scorecard 
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approach,  benefits  ean  be  broken  down  into  four  eategories:  finaneial  perspective, 
eustomer  perspeetive,  internal  perspeetive,  and  innovation  &  learning  (Table  2.1). 

Clearly  an  absolute  definitive  list  of  knowledge  management  benefits  has  yet  to 
be  identified.  However,  four  common  categories  of  KM  benefits  do  seem  to  emerge: 
internal  or  business  benefits,  eustomer  benefits,  employee  benefits,  and  finaneial  benefits 
(Lopez  &  Rayboum,  2003;  North  &  Hornung,  2003). 


Financial  Perspective 

Customer  Perspective 

•  Increased  Sales 

•  Customer  satisfaction 

•  Improved  Productivity 

•  Fewer  Returns 

Internal  Perspective 

Innovation  &  Learning 

•  Cycle  Time  Reduction 

•  Faster  to  Competence 

•  Community  Participation 

•  Employee  Satisfaction 

Table  2.1  -  APQC  KM  benefits  balanced  seoreeard 


Importance  of  Identifying  Knowledge  Management  Benefits 

It  is  important  for  organizations  to  identify  the  benefits  provided  by  KM  for 
several  reasons.  Murray  (2002)  argues  that  in  order  to  suecessfully  eapitalize  on  KM,  an 
organization  must  first  deeide  whieh  partieular  benefits  it  desires  as  the  outeome. 
Unfortunately,  little  progress  has  been  made  with  regards  to  aeeurately  estimating  the 
benefits  KM  may  provide  to  an  organization  (Firestone,  2001).  In  faet,  a  predominant 
belief  is  that  simply  aequiring  and  fostering  the  right  knowledge  will  produee  outcomes 
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that  will  automatically  benefit  the  organization  (Murray,  2002).  In  other  words,  not 
everyone  is  really  sure  what  KM  is  doing  for  them,  but  if  the  right  knowledge  is  there,  the 
benefits  must  be  there  also.  When  companies  do  decide  to  estimate  the  benefits  of  a  KM 
initiative,  often  there  is  simply  the  creation  of  an  intuitive  list  of  outcomes  with  the 
assumption  that  they  will  be  unequivocal  benefits  (Firestone,  2001).  No  attempt  is  made 
to  tie  the  envisioned  outcomes  of  KM  to  eorporate  goals  or  business  processes  (Firestone, 
2001).  Perhaps  it  is  not  surprising  that  many  organizations  that  have  implemented  KM 
ultimately  have  trouble  deriving  any  useful  benefits  (Murray,  2002). 

Measuring  the  Benefits  of  Knowledge  Management 

Another  reason  for  identifying  KM  benefits  is  to  set  the  stage  for  a  proeess  of 
measuring  those  benefits.  Organizational  leaders  naturally  want  proof  that  the  money 
they  are  investing  into  a  new  KM  initiative  is  going  to  produee  a  measurable  result 
(Murray,  2002;  Skyrme  &  Amidon,  1998).  Unfortunately,  one  of  the  problems  with  KM 
is  that  the  biggest  benefits  are  often  by  their  very  nature  immeasurable  (Skyrme  & 
Amidon,  1998).  After  all,  how  can  the  value  of  knowledge  contained  within  an 
individual’s  mind  be  measured  quantitatively?  This  is  especially  true  in  business,  where 
the  majority  of  measures  are  financial  in  nature  (Skyrme  &  Amidon,  1998).  Again,  it  is 
diffieult  to  put  a  dollar  value  on  knowledge. 

Taking  steps  toward  measuring  the  benefits  of  KM  at  a  minimum  first  requires 
identification  of  those  benefits,  as  it  is  impossible  to  measure  something  that  eannot  be 
identified.  This  problem  is  reeognized  across  many  organizations  who  are  praetieing 
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KM,  and  has  been  identified  in  the  aeademie  KM  community  as  well.  In  a  recent  study 
conducted  by  Edwards,  Handzic,  Carlsson,  and  Nissen  (2003),  finding  and  measuring  the 
business  benefits  of  KM  was  identified  as  a  key  issue  for  research. 

Benefits  of  Knowledge  Management  in  the  DoD 

Civilian  organizations  are  not  unique  in  their  need  to  identify  as  well  as  measure 
the  business  benefits  of  KM.  Showing  a  return  on  investment  of  KM  initiatives  in  the 
DoD  is  just  as  necessary  to  gain  and  keep  leadership  support  (Bartczak,  2002;  Bennet  & 
Porter,  2003;  DoD/CIO,  2001).  While  a  list  of  KM  benefits  can  be  generated  from  a 
commercial  perspective,  the  same  list  would  not  necessarily  apply  from  a  DoD 
perspective.  As  North  and  Homung’s  (2003)  study  suggests,  the  benefits  knowledge 
management  provided  to  an  organization  depended  upon  the  KM  approach  taken. 
Ultimately  each  organization  must  generate  a  list  of  benefits  appropriate  to  their  specific 
business  objectives  (Firestone,  2001). 

As  of  yet,  no  studies  have  been  conducted  with  the  goal  of  identifying  the  specific 
benefits  that  the  DoD  should  expect  from  practicing  KM.  As  a  first  step,  this  thesis 
attempts  to  build  a  cursory  list  of  KM  benefits  that  are  already  being  realized  through 
some  of  the  KM  efforts  present  across  the  DoD. 
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III.  METHODOLOGY 


Introduction 

The  goal  of  this  research  is  to  identify  the  realized  benefits  of  KM  in  the  DoD. 
While  several  attempts  at  such  research  have  been  made  in  the  commercial  sector,  there 
is  a  lack  of  research  in  this  arena  that  takes  into  account  the  unique  DoD  perspective.  For 
this  reason,  it  appeared  a  Delphi  study  would  be  an  appropriate  starting  point  for 
examination  of  the  issue.  Lindstone  and  Turoff  (2002)  note  that  the  Delphi  method  is  a 
particularly  valid  choice  when  the  problem  does  not  lend  itself  to  precise  analytical 
techniques  but  can  benefit  from  subjective  judgments  on  a  collective  basis. 

Overview  of  Methodology 

The  first  stage  of  research  for  this  thesis  began  with  a  literature  review  in  order  to 
identify  the  issue  and  develop  the  subject  background.  A  list  of  KM  benefits  pulled  from 
existing  research  on  commercial  organizations  was  then  compiled.  During  the  second 
stage,  a  formal  study  was  conducted  to  validate  this  list  of  benefits  from  a  strictly  DoD 
perspective.  The  Delphi  method  was  used  to  identify  a  consensus  among  a  group  of  DoD 
KM  experts  regarding  which  benefits  were  key  in  a  DoD  context.  Results  were  analyzed 
in  the  final  stage,  and  recommendations  developed. 

Delphi  Method 

The  Delphi  method  is  a  structured,  multipass  group  decision  process  developed  to 
address  research  problems  where  there  is  no  rigid  answer  (Keil,  Tiwana,  &  Bush,  2002). 
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It  is  designed  to  draw  on  a  group  of  experts  to  solve  a  problem  in  a  specific  area  while 
minimizing  the  negative  effects  of  group  interaction  (Rowe  &  Wright,  1999).  Key 
provisions  of  the  Delphi  method  are: 

■  Feedback  of  individual  contributions  of  information  and  knowledge 

■  Assessment  of  the  group  judgment  or  view 

■  Opportunity  for  individuals  to  revise  views 

■  A  degree  of  anonymity  for  the  individual  responses 

(Lindstone  &  Turoff,  2002) 

The  Delphi  method  is  quite  flexible,  and  can  be  applied  any  number  of  different  ways  as 
long  as  the  key  provisions  are  incorporated.  According  to  Lindstone  and  Turoff  (2002), 
“if  anything  is  ‘true’  about  Delphi  today,  it  is  that  in  its  design  and  use  Delphi  is  more  of 
an  art  than  science.” 

Application  of  the  Delphi  method  for  this  research  effort  was  adapted  from  Keil  et 
al.’s  (2002)  study  of  user  and  project  manager  perceptions  of  IT  risk.  The  initial  step  of 
this  particular  study  was  to  select  a  panel  of  domain  experts.  For  the  first  round  of 
research,  each  expert  was  presented  with  a  list  of  issues  related  to  the  research  problem 
and  asked  to  rate  the  issues  in  terms  of  significance.  Data  were  collated  and  an  overall 
consensus  for  each  rating  was  computed  using  the  mean  as  well  as  standard  deviation  for 
each  response.  For  the  second  and  final  round,  panelists  were  presented  with  the  issues. 
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along  with  the  group  consensus  measurements.  At  this  point,  each  expert  was  given  an 
opportunity  to  review  their  ratings  in  light  of  the  group  response. 

Delphi  Limitations 

Delphi  should  not  be  used  unless  three  critical  conditions  are  present.  First, 
adequate  time  must  be  available  to  thoroughly  conduct  the  study,  which  is  generally 
estimated  at  a  minimum  of  45  days.  Second,  participants  must  be  knowledgeable  and 
able  to  clearly  communicate  their  ideas.  Third,  a  high  degree  of  motivation  is  needed  to 
offset  the  tendency  for  participants  to  drop  out  as  the  study  progresses  (Delbecq,  Van  De 
Ven,  &  Gustafson,  1975).  Lindstone  and  Turoff  (2002)  note  the  potential  problem  of 
participant  dropout  as  well,  and  suggest  that  failure  to  allow  participants  to  contribute 
their  own  opinions  and  perspectives  can  negatively  impact  results.  Finally,  Lang  (1998) 
warns  of  the  problem  of  bias  in  Delphi  studies  that  can  result  from  poorly  worded  or 
leading  questions. 

Several  steps  were  taken  to  counteract  these  limitations  for  this  study.  First,  a 
period  of  two  months  was  set  aside  to  conduct  the  Delphi  research  in  order  to  give  ample 
time  for  survey  responses.  Second,  participants  were  selected  from  a  pool  of  experts  who 
have  had  at  least  one  year  of  experience  implementing  knowledge  management  initiatives 
in  the  DoD,  which  increased  the  likelihood  that  they  had  both  extensive  knowledge  of  the 
subject  and  an  ability  to  communicate  ideas.  Third,  respondents  were  assumed  to  be 
sufficiently  motivated  due  to  their  decision  to  voluntarily  participate  in  the  study.  Panel 
members  were  solicited  via  an  open  invitation  letter  which  clearly  identified  participation 


as  optional.  In  addition,  panel  members  were  given  an  opportunity  to  provide  comments 
regarding  each  potential  benefit  and  how  it  applies  specifically  to  their  organization 
within  the  DoD.  This,  as  well  as  the  opportunity  to  answer  an  open-ended  question  at 
the  end  of  each  questionnaire,  follows  Lindstone  and  Turoff  s  suggestion  to  avoid  a 
negative  impact  on  results  by  allowing  the  participants  to  contribute  personal  opinions. 
Finally,  the  bias  problem  related  to  poorly  worded  or  leading  questions  identified  by 
Lang  was  avoided  since  no  direct  questions  were  asked  in  the  Delphi  portion  of  the 
survey.  Respondents  were  simply  presented  with  an  initial  list  of  items  and  asked  to  rate 
the  degree  of  benefit  for  each. 

Delphi  Committee  Development  and  Participant  Selection 

Selecting  the  appropriate  members  of  the  Delphi  panel  is  the  key  to  successful 
application  of  the  method  (Gordon,  1992).  Since  results  of  the  study  are  entirely 
dependent  upon  the  expertise  and  knowledge  of  the  panel  members,  it  is  essential  to 
select  experts  in  a  manner  consistent  with  the  goals  of  the  research  effort.  For  this 
research,  selection  criteria  was  consistent  with  Holsapple  and  Joshi’s  (2000)  Delphi  study 
identifying  KM  influence  factors.  The  panel  consisted  of  Department  of  Defense 
personnel  having  an  active  track  record  in  KM  leadership  and/or  practice.  In  addition, 
each  member  has  had  at  least  two  years  of  experience  working  with  KM.  In  order  to  take 
into  account  both  an  overall  strategic  as  well  as  a  more  directly  observed  perspective  on 
KM  benefits,  care  was  taken  to  include  both  those  who  are  leading  the  KM  charge  in  the 
DoD  as  well  as  those  who  have  been  directly  involved  with  the  implementation  of  KM 


projects.  To  ensure  applicability  across  the  entire  organization,  as  well  as  account  for 
variations  across  services,  it  was  essential  to  solicit  input  from  all  military  branches  as 
well  as  the  overall  DoD  level. 

The  final  Delphi  panel  consisted  of  both  leaders  and  practitioners  in  the  Army, 
Navy,  Air  Force,  Defense  Logistics  Agency,  and  the  Defense  Information  Services 
Agency.  In  order  to  mitigate  the  differing  viewpoints  of  leadership  versus  practitioners, 
input  was  garnered  from  several  high-level  leaders  involved  in  KM  (CIO  level).  The 
remaining  members  had  experience  as  a  practitioner  implementing  at  least  one  large- 
scale  knowledge  management  project  for  their  respective  organization.  An 
organizational  breakdown  of  the  Delphi  panel  is  provided  in  table  3.1. 


ORGANIZATION 

DELPHI  PARTICIPANTS 

Army 

4 

Navy 

I 

Marines 

I 

Air  Force 

I 

Defense  Logistics  Agency 
(DLA) 

I 

Defense  Information  Services 
Agency  (DISA) 

3 

Table  3.1  -  Delphi  panel  breakdown  by  organization 


Initial  invitations  to  participate  in  the  study  were  submitted  to  a  group  of 
individuals  identified  through  the  DoD  Knowledge  Management  community  of  practice 
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(CoP)  located  at  http://dodkm.communispace.com.  Membership  in  this  eommunity  of 
practice  is  reserved  for  personnel  who  have  a  direet  impact  upon  KM  activities  across  the 
DoD,  which  made  it  an  excellent  resource  to  locate  potential  panel  members  who  met  the 
eriteria  for  this  study.  An  initial  list  of  possible  members  was  compiled  by  reading  the 
member  biographies  loeated  at  the  CoP  website  and  sereening  for  individuals  who  might 
fit  the  study  eriteria.  Invitations  to  participate  in  the  study  were  sent  whieh  identified  the 
need  for  partieipants  who  had  at  least  one  year  of  experience  working  with  KM  and  were 
currently  overseeing  or  working  with  KM  in  their  organizations. 

A  total  of  1 1  individuals  were  finally  selected  for  the  eommittee,  which  is 
consistent  with  other  Delphi  studies.  There  is  no  clear  eonsensus  on  the  number  of 
individuals  required  for  a  Delphi  panel,  though  Helmer  and  Dalkey  (1983)  used  a  panel 
of  only  seven  experts  in  their  original  Delphi  experiment  of  1953.  Clayton  (1997)  found 
that  a  Delphi  panel  consisting  of  a  homogeneous  group  of  experts  from  the  same 
diseipline  (i.e.  all  electrical  engineers)  needs  15-30  people,  while  a  more  heterogeneous 
group  (i.e.  experts  from  the  same  area,  but  different  social/professional  levels)  needs  only 
5-10  members.  The  1 1  members  of  this  Delphi  panel  consisted  of  a  heterogeneous  group 
of  DoD  KM  experts  representing  both  the  leadership  as  well  as  practitioner  levels,  which 
should  allow  for  an  adequate  diversity  of  inputs.  A  list  of  nine  participants  along  with 
their  organization  and  position  as  well  as  a  summary  of  the  KM  systems/programs  they 
are  responsible  for  can  be  found  in  Appendix  A.  Two  participants  chose  not  to  divulge 
any  personal  information. 
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Consensus 


One  of  the  goals  of  the  Delphi  process  is  to  identify  a  consensus  among  the  panel 
members  regarding  the  issues.  In  fact,  when  the  method  was  first  developed  in  the 
1950’s,  establishing  consensus  was  seen  as  the  primary  goal  (Scheibe,  Skutsch,  & 
Schofer,  1975).  However,  as  time  passed  and  the  methodology  matured,  non-consensus 
was  also  seen  as  a  valid  result  of  the  Delphi  methodology,  as  was  non-consensus  with  a 
trend  toward  consensus  (Lindstone  &  Turoff,  2002). 

For  this  particular  application  of  the  Delphi,  identifying  consensus  as  well  as  a 
trend  toward  consensus  were  both  desired  outcomes.  Knowledge  management  is  still  a 
new  and  emerging  practice,  especially  within  the  DoD.  Given  the  diversity  of 
organizational  goals  across  the  DoD,  as  well  as  the  differing  levels  of  KM  maturity,  it  is 
highly  unlikely  that  a  heterogeneous  group  of  experts  would  come  to  a  consensus  on  each 
and  every  KM  benefit. 

Several  methods  of  measuring  consensus  have  been  identified  in  the  literature 
(Bower,  2001).  Two  examples  are  achieving  a  decrease  in  standard  deviation  between 
rounds  (Dickson,  Leitheiser,  Wetherbe,  &  Nechis,  1984)  and  a  reduction  in  the  variance 
of  participant  responses  over  successive  rounds  (Rowe  &  Wright,  1999).  Scheibe, 
Skutsch,  and  Schofer  (1975)  refer  to  consensus  being  achieved  when  there  is  a  variation 
in  average  response  of  less  than  15%  over  succeeding  rounds.  This  is  also  known  as 
opinion  stability. 

For  this  research  effort,  consensus  and  opinion  stability  measurement  have  been 
adapted  from  the  method  used  by  Bower  (2001)  in  his  research  regarding  the 
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development  of  a  decision  framework  for  KM  projects.  Consensus  will  be  determined  by 
comparing  group  responses  to  a  mean  group  score  for  each  questionnaire  item.  A 
reduction  in  the  standard  deviation  of  the  mean  group  score  through  succeeding  rounds 
will  be  seen  as  movement  towards  consensus  (Rowe  &  Wright,  1999).  For  this  study,  the 
following  definitions  were  used: 


CONSENSUS: 

•  90%  or  more  of  all  respondents’  inputs  (10  of  1 1  total)  fall  within  +/-  1  standard 
deviation  of  the  group  mean.  Fractional  numbers  for  the  upper  and  lower  bounds 
are  rounded  to  the  nearest  whole  number.  Also,  of  the  remaining  10%,  no  more 
than  one  response  can  have  an  overall  opinion  that  conflicts  with  the  group 
response  (i.e.  all  group  responses  but  one  fall  within  the  1-3  range  (generally 
agree  not  a  benefit)  or  4-6  range  (generally  agree  is  a  benefit)  (Bower,  2001). 

MOVE  TOWARD  CONSENSUS: 

•  A  decrease  in  the  standard  deviation  of  an  item’s  response  between  rounds  will 
indicate  a  trend  toward  consensus  for  that  item. 


Again,  although  consensus  of  the  group  response  is  being  measured,  this  is  not  the 
only  function  of  the  Delphi  method  for  this  research.  Identification  of  benefits  where 
consensus  has  yet  to  be  achieved,  but  there  is  evidence  of  a  trend  towards  consensus,  is 
also  important. 

Compilation  of  KM  Benefits 

A  list  of  16  KM  benefits  were  compiled  for  creation  of  the  Delphi  questionnaires. 
The  benefits  fell  into  four  categories  using  the  balanced  scorecard  approach  toward  KM 


benefits,  which  was  outlined  in  chapter  II.  More  specifically,  each  benefit  was  adapted 
from  those  identified  in  North  and  Homung’s  (2003)  KM  benefits  study.  Since  the  DoD 
is  a  non-profit  organization,  benefits  related  to  financial  results  were  removed  for  this 
application.  Of  the  remaining  benefits,  the  business  process  benefit  of  “better  process 
transparency”  and  employee  satisfaction  benefit  of  “increased  personal  market  value” 
were  determined  to  be  non-applicable  to  the  DoD  realm,  and  thus  removed.  The  final  16 
benefits  used  for  the  Delphi  questionnaire  are  listed  in  Table  3.2. 


BUSINESS  PROCESS  BENEFITS 

Acceleration  of  processes 

Avoidance  of  redundancies 

Re-use  of  internal  knowledge 

Reduction  of  errors 

Time  savings  in  doing  routine  work 

CUSTOMER  SATISFACTION  BENEFITS 

Better  response  time 

Improvements  in  product  and  service  quality 

Better  customer  communication 

Increase  in  customer  satisfaction 

Better  customer  retention 

Increased  information  content 

EMPLOYEE  SATISFACTION  BENEFITS 

Increased  motivation 

Improved  teamwork 

Shorter  training  periods 

Development  of  job  skill 

Enhancement  of  personal  knowledge 

Table  3.2  -  KM  benefits  identified  from  literature  for  Delphi  research 


It  is  important  to  remember  that  these  benefits  have  been  validated  in  the 
literature  as  being  applieable  to  eommereial  organizations.  The  goal  of  this  Delphi 
research  is  to  determine  their  potential  applicability  to  KM  in  the  DoD. 


IV.  RESULTS  AND  ANALYSIS 


Introduction 

This  chapter  will  begin  with  a  quiek  summary  of  the  Delphi  proeess  used  for  this 
researeh  effort,  ineluding  a  quick  overview  of  the  questionnaire  submission  and  response 
outeomes,  as  well  as  a  short  summary  of  the  results.  A  detailed  analysis  of  eaeh  Delphi 
item  will  follow,  including  consensus  and  opinion  stability  measurements,  as  well  as  a 
summary  of  eomments  submitted  by  the  eommittee  members.  A  discussion  on  the  group 
response  regarding  the  diffieulty  of  measuring  KM  benefits  is  presented  next,  followed 
by  a  final  review  of  the  research  questions. 

First  Round:  Questionnaire  One 

Partieipants  were  asked  to  examine  the  list  of  16  KM  benefits  and  rate  each  item 
based  upon  a  6-point  Likert-type  seale  (1  =  no  benefit,  6  =  strong  benefit).  Each  member 
was  also  asked  to  provide  comments  regarding  eaeh  potential  benefit’s  applieability  to 
his/her  speeifie  organization.  Linally,  the  respondents  were  asked  an  open-ended 
question  designed  to  identify  any  KM  benefits  they  had  observed  in  their  organizations 
that  were  not  present  in  the  list  provided. 

Lor  both  the  first  and  seeond  rounds,  the  survey  was  sent  as  a  Microsoft  Word 
attachment  via  eleetronie  mail.  Respondents  were  asked  to  return  the  survey  via  e-mail 
within  14  days  of  reeeiving  the  questionnaire.  Refer  to  Appendix  B  for  a  copy  of 
questionnaire  one. 


Four  round  one  questionnaires  were  returned  within  five  days  of  submission  to  the 
panel,  and  three  more  were  returned  before  the  14-day  deadline  had  been  reached.  A 
follow-up  e-mail  was  sent  to  the  remaining  panel  members,  and  all  round  one 
questionnaires  were  returned  within  the  next  five  days.  Six  items  achieved  consensus  in 
the  first  round. 

Second  Round:  Questionnaire  Two 

In  the  second  and  final  questionnaire,  panel  members  were  presented  with  a  list  of 
benefits  that  did  not  achieve  consensus  in  round  one,  along  with  the  comments  provided 
by  each  panel  member.  Members  were  asked  to  review  their  round  one  response  for  each 
benefit,  taking  into  consideration  the  group  mean  response  as  well  as  the  group 
comments.  For  each  item,  the  respondent  could  either  keep  their  initial  rating  or  change 
it  to  better  agree  with  group  consensus.  An  explanation  was  required  for  any  rating  that 
didn’t  coincide  with  the  group  mean.  In  other  words,  a  mean  rating  of  4  or  above  would 
require  a  respondent  to  explain  an  individual  rating  of  3  or  below.  Similarly,  a  mean 
rating  of  3  or  below  would  require  an  explanation  for  a  rating  of  4  or  above.  Finally, 
panelists  were  asked  a  single  open-ended  question  designed  to  identify  any  problems  they 
might  experience  measuring  KM  benefits  in  their  organization.  Refer  to  Appendix  C  for 
a  copy  of  questionnaire  2. 

Eight  questionnaires  were  returned  before  the  15  Dec  03  due  date.  Reminders 
were  sent  to  each  of  the  remaining  three  panel  members,  however  no  additional  responses 
were  received  until  after  the  holiday  break.  Two  more  questionnaires  were  returned  by  5 


Jan  04.  One  participant  was  unavailable  for  a  great  length  of  time  and  was  unable  to 
return  the  final  round  two  questionnaire  until  20  Jan  04.  However,  this  resulted  in  a  100 
percent  response  rate  with  all  1 1  round  one  and  two  questionnaires  returned  for  analysis. 

Responses  to  questionnaires  one  and  two  were  analyzed  to  determine  group 
consensus.  The  mean  benefit  rating  from  each  participant  was  computed,  along  with  the 
standard  deviation.  Consensus  was  achieved  if  at  least  10  of  1 1  ratings  fell  within  one 
standard  deviation  of  the  mean  rating.  For  those  items  that  did  not  achieve  consensus 
after  round  two,  a  trend  toward  consensus  was  calculated  by  comparing  the  standard 
deviation  of  the  mean  responses  between  the  two  rounds.  A  reduction  in  the  standard 
deviation  of  the  mean  responses  was  seen  as  a  trend  toward  consensus. 

After  the  first  round,  consensus  was  achieved  on  6  of  the  16  items  in  the  Delphi 
questionnaire.  After  round  two,  no  further  consensus  was  achieved  on  the  remaining 
items.  However,  a  trend  toward  consensus  was  measured  on  9  of  the  10  remaining  items 
yet  to  achieve  full  consensus.  The  final  remaining  item  indicated  an  increase  in  the 
standard  deviation  of  the  mean  response  between  rounds,  signaling  a  continual  lack  of 
consensus  among  the  group  on  that  item.  Table  4.1  presents  a  summary  of  the  steps 
taken  in  the  Delphi  process  for  this  research  effort. 


Round  1 

Round  2 

Date  Mailed 

3  Nov  2003 

1  Dec  2003 

Date  Due 

17  Nov  2003 

15  Dec  2003 

Instrument 

Likert  Scale 

Likert  Scale 

Data  Colleeted 

Level  of  impact  for  each  identified 
item,  identification  of  additional 
impacts 

Level  of  impact  for  each  identified  item, 
reason  for  disagreeing  with  consensus, 
suggestions  for  resolving  impacts 

Data  Analysis 

Compute  mean,  standard  deviation 

Compute  mean,  standard  deviation 

Consensus 

Achieved? 

6  of  16  items 

No  further  consensus,  however  move 
toward  consensus  achieved  for  9  of  the 

10  remaining  items 

Table  4.1  -  Summary  of  Delphi  process 


Summary  of  Results 

Analysis  of  the  Delphi  questionnaires  resulted  in  seven  items  being  identified  by 
the  group  as  having  a  benefit  related  to  KM  in  the  DoD.  Nine  more  items  were  identified 
as  not  having  a  benefit,  though  one  of  those  items  failed  to  aehieve  eonsensus  or  a  move 
toward  consensus.  Table  4.2  represents  the  list  of  questionnaire  items  along  with  their 
relevant  benefit  and  consensus  data. 


BUSINESS  PROCESS  BENEFITS 

Benefit 

No  Benefit 

Consensus 

Movement 

Toward 

Consensus 

Acceleration  of  processes 

X 

X 

Avoidance  of  redundancies 

X 

X 

Re-use  of  internal  knowledge 

X 

X 

Reduction  of  errors 

X 

X 

Time  savings  in  doing  routine  work 

X 

X 

BENEFITS  CONCERNING 
CUSTOMER  SATISFACTION 

Benefit 

No  Benefit 

Consensus 

Move 

Toward 

Consensus 

Better  response  time 

X 

X 

Improvements  in  product  and  service 
quality 

X 

X 

Better  customer  communication 

X 

X 

Increase  in  customer  satisfaction 

X 

X 

Better  customer  retention 

X 

X 

Increased  information  content 

X 

X 

BENEFITS  CONCERNING 
EMPLOYEE  SATISFACTION 

Benefit 

No  Benefit 

Consensus 

Move 

Toward 

Consensus 

Increased  motivation 

X 

X 

Improved  teamwork 

X 

X 

Shorter  training  periods 

1 

Mo  consensus  mesasurec 

1 

Development  of  job  skill 

X 

X 

Enhancement  of  personal  knowledge 

X 

X 

Table  4.2  -  Summary  of  Delphi  questionnaire  analysis  results 


Six  items  immediately  achieved  consensus  among  the  1 1  members  of  the  Delphi 
panel  after  the  first  round.  Of  the  six  items,  three  were  agreed  upon  by  the  panel  as 
generally  having  no  benefit  related  to  KM  practice  in  the  DoD.  Again,  for  analysis 
purposes,  an  item  with  a  mean  rating  below  3.5  was  seen  as  having  no  benefit  related  to 
KM  in  the  DoD,  since  it’s  ratings  were  generally  at  the  low-end  of  the  scale.  An  item 
with  a  mean  rating  above  3.5  was  seen  as  having  a  benefit,  since  the  majority  of  its 
ratings  were  at  the  upper-end  of  the  scale.  The  items  along  with  their  mean  ratings  are 
identified  in  Table  4.3  below. 


Potential  KM  Benefit 

Mean  Rating  I 

Avoidance  of  redundancies 

3.18 

Reduction  of  errors 

3.27 

Better  customer  retention 

2.18 

Table  4.3  -  Consensus  items  identified  as  NOT  A  BENEFIT  related  to  KM 


The  other  three  items  that  achieved  consensus  were  identified  by  the  panel  as  a 
general  benefit  related  to  KM  practice  in  the  DoD  (mean  rating  above  3.5).  The  items 
along  with  their  mean  rating  are  presented  in  Table  4.4  below. 


Potential  KM  Benefit 

Mean  Rating  I 

Time  savings  in  doing  routine  work 

4.09 

Increased  information  content 

4.27 

Improved  teamwork 

3.91 

Table  4.4  -  Consensus  items  identified  as  a  BENEFIT  related  to  KM 


The  remaining  10  items  failed  to  aehieve  eonsensus  after  the  second  round.  Of 


the  10  remaining  items,  9  achieved  a  reduction  in  the  standard  deviation  of  the  mean 
rating  from  round  one  to  round  two.  This  indicates  a  move  toward  consensus,  since  it 
signals  that  the  variation  among  the  ratings  being  submitted  by  the  panel  members  is 
getting  smaller.  In  other  words,  the  ratings  are  starting  to  center  upon  a  common  number. 
The  items  along  with  their  mean  rating  standard  deviations  in  round  one  and  round  two 
are  identified  in  Table  4.5  below. 


Potential  KM  Benefit 

Round  1 
Std  Dev 

Round  2 
Std  Dev 

Acceleration  of  processes 

1.48 

1.34 

Re-use  of  internal  knowledge 

1.34 

1.08 

Better  response  time 

1.57 

1.49 

Improvements  in  product  service  and  quality 

1.66 

1.51 

Better  customer  communication 

1.54 

1.38 

Increase  in  customer  satisfaction 

1.12 

0.81 

Increased  motivation 

1.57 

1.25 

Development  of  job  skill 

1.51 

1.30 

Enhancement  of  personal  knowledge 

1.80 

1.42 

Table  4.5  -  Items  indicating  a  move  toward  consensus 


Four  of  the  nine  items  showing  a  move  toward  consensus  featured  a  mean  rating 
above  3.5,  which  according  to  the  definition  presented  earlier  indicates  the  panel 
identified  them  as  having  a  benefit  related  to  KM  in  the  DoD  (Table  4.6).  The  five 
remaining  items  earned  a  mean  rating  of  3.5  or  below,  indicating  a  general  belief  that 
they  are  likely  not  a  benefit  related  to  KM  in  the  DoD  (Table  4.7). 
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Potential  KM  Benefit 

Mean  Rating 

Acceleration  of  processes 

4.00 

Re-use  of  internal  knowledge 

4.18 

Increased  motivation 

3.82 

Enhancement  of  personal  knowledge 

4.27 

Table  4.6  -  Items  moving  toward  consensus  and  identified  as  a  BENEFIT  related  to  KM 


Potential  KM  Benefit 

Mean  Rating 

Better  response  time 

3.27 

Improvements  in  product  and  service  quality 

2.91 

Better  customer  communication 

3.09 

Increase  in  customer  satisfaction 

2.36 

Development  of  job  skill 

3.09 

Table  4.7  -  Items  moving  toward  consensus  but  identified  as  NOT  A  BENEFIT  related  to 

KM 


The  final  remaining  item,  development  of  job  skill,  failed  to  achieve  consensus  or 
indicate  a  movement  toward  consensus.  The  standard  deviation  of  the  mean  rating  for 
this  item  increased  from  1.41  in  round  one  to  1.49  in  round  two,  while  the  mean  rating 
increased  from  3.00  in  round  one  to  3.27  in  round  two.  Further  detail  of  what  this  might 
mean  is  presented  later  in  the  next  section  of  this  thesis,  which  features  an  in-depth 
analysis  of  each  item  and  incorporates  the  written  comments  submitted  by  each  panel 
member. 


Analysis  of  Individual  Item  Responses 

The  following  section  details  the  response  for  each  benefit  contained  in  the  Delphi 
questionnaires.  Each  item  will  follow  the  same  format,  beginning  with  whether  the 


benefit  reached  consensus  or  opinion  stability,  as  well  as  a  table  detailing  the  actual 
consensus  calculations  for  each  item.  A  summary  of  the  written  responses  provided  by 
each  panel  member  will  also  be  presented.  For  analysis  purposes,  a  mean  benefit  rating 
below  3.5  for  an  item  indicates  that  item  was  generally  not  considered  to  be  a  benefit 
related  to  KM.  Conversely,  a  mean  benefit  rating  above  3.5  indicates  the  item  was 
generally  considered  to  be  a  benefit.  A  comprehensive  list  of  comments  supplied  by  each 
panel  member  can  be  obtained  by  reviewing  the  round  two  questionnaire  in  Appendix  C. 

Business  Process  Benefits 

These  items  focused  on  benefits  of  KM  specifically  related  to  the  business 
processes  of  an  organization.  Three  of  the  items  achieved  consensus  after  the  first  round, 
while  the  remaining  two  showed  a  movement  toward  consensus  after  the  second  round. 

Item  1:  Acceleration  of  processes 

This  item  did  not  achieve  a  consensus  after  round  two.  However,  a 
decrease  in  the  standard  deviation  of  the  mean  response  between  rounds  was 
measured,  signaling  a  trend  toward  consensus.  The  mean  response  after  round 
two  was  4.00,  which  indicates  acceleration  of  business  processes  was  generally 
considered  to  be  a  benefit  related  to  KM  in  the  DoD.  Table  4.8  lists  the  ratings 
and  consensus  calculations  for  this  item. 
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Acceleration  of  Proeesses 


Round  1  Rating 

Round  2 
Rating 

Respondent  1 

6 

6 

Respondent  2 

6 

5 

Respondent  3 

5 

5 

Respondent  4 

4 

4 

Respondent  5 

2 

2 

Respondent  6 

5 

5 

Respondent  7 

2 

2 

Respondent  8 

3 

3 

Respondent  9 

5 

5 

Respondent  10 

3 

3 

Respondent  1 1 

3 

4 

Mean 

4.00 

4.00 

Std  Deviation 

1.48 

1.34 

Upper  /  Lower  Bound 

5.48-2.52 

5.34-2.66 

BENEFIT 

Table  4.8  -  Final  ratings  and  eonsensus  ealculations  for  acceleration  of 

processes 


Seven  of  the  1 1  respondents  submitted  a  final  rating  for  this  benefit  of  at  least  a 
four  or  above.  For  these  respondents,  the  general  consensus  seemed  to  be  that  having  a 
KM  system  allows  people  to  spend  less  time  looking  for  information,  whieh  means  more 
time  spent  producing  results.  One  member  eommented,  “With  the  implementation  of 
KM  we  have  been  able  to  automate  and  streamline  several  reporting  proeesses.  The  data 
is  more  aeeurate  and  timely.”  Another  noted  that  KM  had  allowed  his  organization  to 
move  away  from  PowerPoint  methods  of  sharing  information  into  a  database  driven 
approach.  This  facilitated  an  automation  of  proeesses  that  had  previously  been  manually 
performed.  Still  another  was  eareful  to  point  out  that  although  KM  had  indeed 


41 


streamlined  processes,  in  some  cases  it  merely  “sped  up  the  mess”  and  automated  a 
process  that  was  not  efficient  in  the  first  place. 

The  remaining  four  respondents  submitted  a  rating  of  3  or  helow,  indicating  they 
generally  helieved  acceleration  of  business  process  was  not  a  benefit  related  to  KM  in  the 
DoD.  One  respondent  felt  that,  while  KM  should  indeed  have  process  benefits,  she 
hadn’t  seen  them  yet  in  her  organization.  She  elaborates: 

“Everyone  wants  KM  and  collaboration  but  they  don’t  know  what  they 
want  to  do  with  it.  They  just  know  it’s  the  hot  thing.  There  are  certainly 
some  early  adopters  who  are  driving  process  change,  but  by  and  large,  I 
perceive  people  feel  they  don’t  have  the  time  or  money  to  devote  to 
changing  the  process.  This  takes  a  lot  of  hand-holding  to  change  the 
culture.” 

Another  panel  member  indicated  that  his  organization’s  current  KM  initiative  was 
too  early  in  the  process  to  claim  success,  but  anticipated  that  process  impact 
would  soon  be  as  positive  as  some  of  the  other  panel  members  reported. 

Item  2:  Avoidance  of  redundancies 

This  item  achieved  an  immediate  consensus  at  the  completion  of  round 
one.  The  overwhelming  majority  of  respondents  submitted  a  rating  of  2  or  3  for 
this  particular  benefit,  with  one  outlier  feeling  this  item  was  a  strong  benefit 
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warranting  a  6  rating.  Table  4.9  lists  the  ratings  and  consensus  calculations  for 
this  item. 

Comments  from  the  majority  of  the  group  indicated  that  their  KM  initiative  was 
still  not  yet  embedded  enough  into  the  organization  to  see  avoidance  of  redundancies  as  a 


Avoidance  of  Redundancies 


Round  1  Rating  Round  2 

Rating 


Respondent  1 

6 

N/A 

Consensus 
achieved  in 
first  round 

Respondent  2 

3 

Respondent  3 

3 

Respondent  4 

4 

Respondent  5 

3 

Respondent  6 

2 

Respondent  7 

3 

Respondent  8 

2 

Respondent  9 

3 

Respondent  10 

3 

Respondent  1 1 

3 

Mean 

3.18 

N/A 

Std  Deviation 

1.08 

Upper  /  Lower  Bound 

4.26-2.10 

NOT  A  BENEFIT 

Table  4.9  -  Final  ratings  and  consensus  calculations  for  avoidance  of  redundancies 


benefit.  However,  many  indicated  that  they  did  believe  KM  would  eventually  provide 
this  benefit.  According  to  one  member: 


“We  have  made  improvements  in  this  area.  However  since  we  are  still  in 
the  pilot  phase  of  our  implementation,  some  redundancy  still  exists. 
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However  as  more  and  more  people  beeome  aware  and  comfortable  with 
the  new  tools/capahilities,  they  are  becoming  more  comfortable  with 
moving  away  from  their  manual/redundant  processes.” 

Another  simply  stated  that,  “The  potential  exists,  hut  I  don’t  think  KM  is 
sufficiently  embedded  for  people  to  maximize  its  use.”  The  individual  who 
believed  avoidance  of  redundancies  was  a  strong  benefit  commented  that  his 
organization  regularly  finds  people  using  the  KM  tools  and  expertise  instead  of 
reinventing  solutions. 

Item  3:  Re-use  of  internal  knowledge 

This  item  did  not  achieve  a  consensus  after  round  two.  However,  a 
decrease  in  the  standard  deviation  of  the  mean  response  between  rounds  was 
measured,  signaling  a  trend  toward  consensus.  The  mean  response  after  round 
two  was  4.18,  which  indicates  re-use  of  internal  knowledge  was  generally 
considered  to  be  a  benefit  related  to  KM  in  the  DoD.  Table  4. 10  lists  the  ratings 
and  consensus  calculations  for  this  item. 

Nine  of  the  1 1  respondents  submitted  a  final  rating  of  4  or  above  for  this  item. 
The  general  consensus  among  the  panel  members  was  that  because  of  the  KM  system  in 
place,  people  were  re-using  what  others  had  done  rather  than  “re-inventing  the  wheel.” 
On  respondent  noted  that,  “the  most  difficult  aspect  to  overcome  was  getting  people  to 
contribute.  KM  does  not  seem  to  flourish  unless  tied  to  performance  appraisals.” 
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Of  the  two  respondents  that  submitted  a  rating  of  3  or  below  for  this  item,  one 
commented  that,  “If  anything,  our  organization  is  falling  further  behind  in  re-using 
knowledge  as  the  flood  of  experience  from  Southwest  and  Central  Asia  overwhelms  our 
current  system  for  digesting  them.”  In  other  words,  the  ability  to  re-use  internal 
knowledge  may  be  severely  hampered  if  a  particular  KM  system  does  not  have  the 
capacity  to  handle  the  level  of  input. 


Re-use  of  Internal  Knowledge 


Round  1  Rating 

Round  2 
Rating 

Respondent  1 

6 

6 

Respondent  2 

4 

4 

Respondent  3 

5 

5 

Respondent  4 

5 

5 

Respondent  5 

5 

4 

Respondent  6 

4 

4 

Respondent  7 

5 

5 

Respondent  8 

3 

3 

Respondent  9 

2 

4 

Respondent  10 

2 

2 

Respondent  1 1 

3 

4 

Mean 

4.00 

4.18 

Std  Deviation 

1.34 

1.08 

Upper  /  Lower  Bound 

5.34-2.66 

5.26-3.10 

BENEFIT 

Table  4.10  -  Final  ratings  and  consensus  calculations  for  re-use  of  internal 

knowledge 


Item  4:  Reduction  of  errors 

This  item  achieved  consensus  at  the  end  of  round  one  with  a  mean  score  of 
3.27,  indicating  that  reduction  of  errors  was  generally  not  considered  to  be  a 


benefit  related  to  KM  in  the  DoD.  Five  of  eleven  respondents  did  submit  a  rating 
of  4  or  above  however,  finding  that  “as  we  re-use  internal  knowledge  more 
effectively,  errors  naturally  decline.  Table  4.1 1  lists  the  ratings  and  consensus 
calculations  for  this  item. 


Reduction  of  Errors 

Round  1  Rating 

Round  2 
Rating 

Respondent  1 

5 

N/A 

Consensus 
achieved  in 
first  round 

Respondent  2 

4 

Respondent  3 

4 

Respondent  4 

4 

Respondent  5 

2 

Respondent  6 

1 

Respondent  7 

5 

Respondent  8 

3 

Respondent  9 

3 

Respondent  10 

2 

Respondent  1 1 

3 

Mean 

3.27 

N/A 

Std  Deviation 

1.27 

Upper  /  Lower  Bound 

4.54-2.00 

NOT  A  BENEFIT 

Table  4.1 1  -  Final  ratings  and  consensus  calculations  for  reduction  of 

errors 


The  remaining  six  respondents  were  not  so  positive  that  their  KM 
implementations  were  reducing  errors  in  their  organization,  thus  resulting  in  the  mean 
rating  below  3.5.  Unfortunately,  many  of  these  respondents  in  the  1  to  3  range  elected 
not  to  comment  on  their  ratings.  Of  the  two  that  did,  one  indicated  that  he  didn’t  find 
excessive  errors  to  be  an  organizational  problem  in  the  first  place.  The  other  noted  that. 


“slow  implementation  of  cutting-edge  KM  processes,  structures,  tools,  and  mindsets  has 


hampered  our  potential  for  reducing  errors.” 


Item  5:  Time  savings  in  doing  routine  work 

This  was  another  item  that  achieved  consensus  after  the  first  round.  The 
mean  rating  for  this  item  was  4.09,  indicating  the  panel  felt  time  savings  in  doing 
routine  work  was  indeed  a  henefit  related  to  KM  practice  in  the  DoD.  Table  4. 12 
lists  the  ratings  and  consensus  calculations  for  this  item. 

Seven  of  the  eleven  respondents  gave  this  item  a  rating  of  4  or  above  and 
submitted  some  of  the  strongest  positive  comments  of  all  the  items  in  the  questionnaire. 
One  respondent  commented  specifically  that  they  are  saving  8  hours  a  week  through  their 
KM  initiative,  while  another  stated  that,  “so  far  this  is  the  biggest  plus  for  KM.”  A 
common  theme  running  through  the  positive  comments  was  the  notion  that  KM  helped 
increase  the  efficiency  of  producing  regular  reports.  For  example,  one  member 
commented  that  KM  enabled  teams  to  put  together  daily  briefings  much  faster: 

“Before  they  had  to  wait  on  each  team  to  send  an  e-mail  with  the  latest  file 
attached.  There  was  often  confusion  as  to  which  attachment  was  the  latest 
version.  Then  the  integration  team  had  to  review  all  the  daily 
presentations  and  eliminate  redundancies  and  try  to  synchronize  the  input. 

Now  the  KM  system  provides  version  control  and  a  way  for  others  to 
review  it  and  then  post  comments,  so  it  is  a  collaborative  review  process.” 
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The  remaining  four  respondents  submitted  a  rating  of  3  or  below  and  did  not  feel  that  KM 
in  their  organizations  was  quite  ready  to  provide  any  time  savings  in  doing  routine  work. 


Time  Savings  in  Doing  Routine  Work 


Round  1  Round  2 


Respondent  1 

6 

N/A 

Consensus 
achieved  in 
first  round 

Respondent  2 

6 

Respondent  3 

6 

Respondent  4 

5 

Respondent  5 

4 

Respondent  6 

4 

Respondent  7 

2 

Respondent  8 

2 

Respondent  9 

5 

Respondent  10 

2 

Respondent  1 1 

3 

Mean 

4.09 

N/A 

Std  Deviation 

1.64 

Upper  /  Lower  Bound 

5.73-2.45 

1  BENEFIT  1 

Table  4.12  -  Final  ratings  and  eonsensus  ealculations  for  time  savings  in  doing 

routine  work 


However,  their  eomments  were  once  again  related  to  the  level  of  maturity  and  acceptance 
of  the  KM  system.  One  respondent  stated: 


“Our  best  practices  have  shown  dramatic  time  savings.  The  prime 
community  of  practice. .  .saves  as  much  as  30  percent  for  active 
participants.  While  we  have  a  state-of-the-art  plan  for  networking  the 
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force  with  structured  CoPs,  it  has  not  yet  been  implemented.  Thus,  our 


impact  on  time  savings  is  limited  to  fragments  of  the  force.” 

Similarly,  another  noted  they  had  not  seen  any  time  savings  yet,  but  anticipate  big 
progress  once  their  KM  initiative  is  fully  operational. 

Benefits  Concerning  Customer  Satisfaction 

These  items  centered  on  benefits  related  to  organizational  customer  satisfaction. 
Two  of  the  six  items  in  this  section  achieved  consensus  after  the  first  round,  while  all 
remaining  items  showed  a  movement  toward  consensus  after  the  second  round. 


Item  6:  Better  response  time 

This  item  did  not  achieve  consensus  after  round  two,  however  it  did  show 
a  movement  toward  consensus.  The  mean  rating  for  this  item  after  round  two  was 
3.27,  indicating  that  the  panel  did  not  see  better  response  time  as  a  benefit  related 
to  KM  in  the  DoD.  Table  4. 13  lists  the  ratings  and  consensus  calculations  for  this 
item. 

Five  respondents  submitted  a  rating  of  4  or  above,  believing  that  KM  does  indeed 
provide  a  timelier  response  to  their  customers’  needs.  One  member  indicated  that 
customer  feedback  has  been  positive  regarding  requirements  being  implemented  in  a 
timely  manner,  while  another  noted  a  unit  deploying  overseas  had  indicated  his 
organization’s  KM  system  had  saved  them  significant  time  in  getting  up  to  speed. 


4Q 


According  to  another  respondent,  “Judicious  use  of  KM  has  shown  significant  reduction 
in  cycle  time  for  those  organizations  that  understand  how  to  implement  effectively.  Once 
again,  this  improvement  is  seen  on  a  case-by-case  basis,  not  enterprise-wide.” 


Better  Response  Time 


Round  1  Rating 

Round  2 
Rating 

Respondent  1 

6 

6 

Respondent  2 

4 

4 

Respondent  3 

5 

4 

Respondent  4 

4 

4 

Respondent  5 

3 

3 

Respondent  6 

1 

1 

Respondent  7 

2 

2 

Respondent  8 

2 

2 

Respondent  9 

5 

5 

Respondent  10 

2 

2 

Respondent  1 1 

4 

3 

Mean 

3.45 

3.27 

Std  Deviation 

1.57 

1.49 

Upper  /  Lower  Bound 

5.03-1.88 

4.76-1.78 

NOT  A  BENEFIT 

Table  4.13  -  Final  ratings  and  consensus  calculations  for  better  response  time 


The  remaining  six  respondents  submitted  a  rating  of  3  or  below,  however  the 
comment  from  one  of  these  individuals  indicates  an  expectation  for  improvement.  He 
writes,  “Fielding  our  Battle  Command  Knowledge  System  (BCKS)  will  transform  this 
into  our  greatest  strength  in  the  next  two  years.”  His  ratings  for  the  rest  of  the  benefits 
concerning  customer  satisfaction  category  are  consistently  low,  with  his  comments 
indicating  the  same  reasoning. 


A  second  individual  submitted  a  1  rating  because  he  saw  his  organization’s  KM 
initiative  as,  “focused  on  the  internal  information  and  knowledge  sharing  needs  of  the 
Agency.”  He  saw  the  KM  system  benefits  passed  onto  customers  only,  “VERY 
indirectly.”  Again,  this  opinion  is  reflected  in  the  individual’s  rating  of  1  for  every  item 
in  the  benefits  eoneerning  customer  satisfaction  category. 

Item  7:  Improvements  in  product  and  service  quality 

Consensus  was  not  achieved  after  round  two  for  this  item;  however,  there 
was  a  decrease  in  the  standard  deviation  of  the  mean  rating  between  rounds 
indicating  a  move  toward  consensus.  The  final  mean  rating  for  this  item  was 
2.91,  indicating  the  panel  did  not  believe  that  improvements  in  product  and 
service  quality  was  a  benefit  related  to  KM  in  the  DoD.  Table  4.14  lists  the 
ratings  and  consensus  calculations  for  this  item. 

Only  two  respondents  submitted  a  final  rating  of  4  or  above  for  this  item,  with  the 
remaining  9  respondents  awarding  a  3  or  below.  This  item  received  the  second-lowest 
final  rating  of  the  16  items. 


Improvement  in  Product  Service  and  Quality 


Round  1  Rating 

Round  2 
Rating 

Respondent  1 

6 

6 

Respondent  2 

5 

3 

Respondent  3 

5 

5 

Respondent  4 

3 

3 

Respondent  5 

3 

3 

Respondent  6 

1 

1 

Respondent  7 

3 

3 

Respondent  8 

2 

2 

Respondent  9 

1 

1 

Respondent  10 

2 

2 

Respondent  1 1 

4 

3 

Mean 

3.18 

2.91 

Std  Deviation 

1.66 

1.51 

Upper  /  Lower  Bound 

4.84-1.52 

4.42-1.40 

NOT  A  BENEFIT 

Table  4.14  -  Final  ratings  and  consensus  calculations  for  improvement  in  product 

and  service  quality 


Item  8:  Better  customer  communication 

Once  again  consensus  was  not  achieved  after  round  two  for  this  item. 

However,  a  decrease  in  the  standard  deviation  of  the  mean  rating  between  rounds 
did  signify  a  move  toward  consensus.  The  panel  awarded  a  final  mean  rating  of 
3.09  to  this  category,  indicating  a  general  disbelief  that  better  customer 
communication  was  a  benefit  of  KM  in  the  DoD.  Table  4.15  lists  the  ratings  and 
consensus  calculations  for  this  item. 

Similar  to  the  last  item,  only  2  of  the  1 1  respondents  submitted  a  mean  rating  of  4 
or  above  while  the  remaining  9  submitted  a  mean  rating  of  3  or  below.  Many  member 
comments  remained  very  similar  to  the  other  items  in  the  benefits  concerning  customer 


satisfaction  group.  These  ineluded  the  expeetation  from  one  individual  that  their  KM 
system  would  provide  this  benefit  once  fully  on-line,  as  well  as  the  opinion  that  the  KM 
system  is  more  for  organizational  than  customer  benefit. 


Better  Customer  Communication 


Round  1  Rating 

Round  2 
Rating 

Respondent  1 

3 

6 

Respondent  2 

5 

3 

Respondent  3 

4 

3 

Respondent  4 

3 

3 

Respondent  5 

1 

2 

Respondent  6 

1 

1 

Respondent  7 

5 

5 

Respondent  8 

2 

3 

Respondent  9 

1 

3 

Respondent  10 

2 

2 

Respondent  1 1 

4 

3 

Mean 

2.82 

3.09 

Std  Deviation 

1.54 

1.38 

Upper  /  Lower  Bound 

4.36-1.28 

4.47-1.72 

NOT  A  BENEFIT 

Table  4.15  -  Final  ratings  and  consensus  calculations  for  better  customer 

communication 


Interestingly,  several  members  granted  a  3  rating  even  though  their  comments  were 
overly  positive.  One  of  these  individuals  noted  an,  “. .  .almost  immediate,  requirements 
and  feedback  times  have  improved,”  while  another  felt  KM  indeed  provided  a  faster 
response  time  to  questions. 


Item  9:  Increase  in  customer  satisfaction 


Consensus  was  not  achieved  on  this  item,  however  a  movement  toward 
consensus  was  observed  after  round  two.  The  final  mean  rating  for  this  item  was 
2.36,  which  happened  to  be  the  second-lowest  rating  granted  by  the  group.  This 
indicates  the  group  generally  did  not  consider  an  increase  in  customer  satisfaction 
to  be  a  benefit  of  KM  in  the  DoD.  This  item  also  had  the  lowest  group  mean 
standard  deviation  of  all  the  items.  Table  4. 16  lists  the  ratings  and  consensus 
calculations  for  this  item. 


Increase  in  Customer  Satisfaction 


Round  1  Rating 

Round  2 
Rating 

Respondent  1 

3 

3 

Respondent  2 

4 

4 

Respondent  3 

4 

3 

Respondent  4 

2 

2 

Respondent  5 

3 

2 

Respondent  6 

1 

1 

Respondent  7 

2 

2 

Respondent  8 

3 

2 

Respondent  9 

1 

2 

Respondent  10 

2 

2 

Respondent  1 1 

4 

3 

Mean 

2.64 

3.09 

Std  Deviation 

1.12 

1.38 

Upper  /  Lower  Bound 

3.76-1.52 

4.47-1.72 

NOT  A  BENEFIT 

Table  4.16  -  Final  ratings  and  consensus  calculations  for  increase  in  customer 

satisfaction 


One  member  submitted  a  final  rating  of  4,  and  was  the  only  group  member  to 
submit  a  final  rating  greater  than  3  as  well  as  the  reason  the  item  did  not  achieve 
consensus.  The  individual’s  comment  attempts  a  defense  of  the  outlier  rating,  stating 
that,  .  .the  KM  portal  and  the  tools  it  offers  seem  to  have  a  high  rate  of  customer 
satisfaction.” 

Of  the  remaining  members  who  submitted  a  3  or  below  rating,  the  general  feeling 
seemed  to  be  that  of  those  who  first  believed  they  had  customers,  satisfaction  was  indeed 
higher.  However,  there  was  no  direct  evidence  available  to  prove  such  an  increase. 


Item  10:  Better  customer  retention 

This  item  achieved  consensus  after  the  first  round.  It  also  featured  the 
lowest  mean  rating  at  2.18,  indicating  that  the  group  generally  felt  it  was  of  no 
benefit  to  KM  in  the  DoD.  Table  4.17  lists  the  ratings  and  consensus  calculations 
for  this  item. 

Two  respondents  submitted  a  rating  of  4  or  above  for  this  item,  with  one 
commenting  that,  “. .  .more  customers  come  to  us  as  we  integrate  our  portal  services  and 
we  haven’t  lost  any.”  The  other  noted  that  as  his  organization  makes  KM  tools  more 
accessible,  “. .  .employees  will  continue  to  find  our  methods  and  support  tools  worth 
choosing.” 

The  overwhelming  majority  of  panel  members  submitted  a  rating  of  3  or 
below  for  this  item.  While  most  chose  not  to  provide  comments  regarding  this 


benefit,  one  did  cite  the  soldier  on  the  field  as  a  customer  and  the  inability  to 
retain  them  despite  making  KM  tools  available. 


Better  Customer  Retention 


Round  1  Round  2 


Respondent  1 

1 

N/A 

Consensus 
achieved  in 
first  round 

Respondent  2 

5 

Respondent  3 

3 

Respondent  4 

2 

Respondent  5 

1 

Respondent  6 

1 

Respondent  7 

2 

Respondent  8 

2 

Respondent  9 

1 

Respondent  10 

2 

Respondent  1 1 

4 

Mean 

2.18 

Std  Deviation 

1.33 

N/A 

Upper  /  Lower  Bound 

3.51-0.85 

NOT  A  BENEFIT 

Table  4.17  -  Final  ratings  and  consensus  calculations  for  better  customer 

retention 


Item  11:  Increased  information  content 

This  was  another  item  that  achieved  consensus  in  the  first  round,  and 
incidentally  featured  a  final  mean  rating  that  tied  for  first  with  item  16,  which  was 
enhancement  of  personal  knowledge.  The  mean  rating  of  4.27  indicated  the  group 
believed  that  increased  information  content  was  a  general  benefit  related  to  KM 
in  the  DoD.  Table  4.18  lists  the  ratings  and  consensus  calculations  for  this  item. 


Increased  Information  Content 


Round  I  Round  2 


Respondent  1 

6 

N/A 

Consensus 
achieved  in 
first  round 

Respondent  2 

6 

Respondent  3 

6 

Respondent  4 

5 

Respondent  5 

3 

Respondent  6 

1 

Respondent  7 

6 

Respondent  8 

2 

Respondent  9 

6 

Respondent  10 

2 

Respondent  1 1 

4 

Mean 

4.27 

N/A 

Std  Deviation 

1.95 

Upper  /  Lower  Bound 

6.23-2.32 

BENEFIT 

Table  4.18  -  Final  ratings  and  consensus  calculations  for  increased  information 

content 


Seven  of  the  1 1  respondents  submitted  a  rating  of  4  or  above,  with  five  submitting 
a  max  rating  of  6.  Group  comments  indicated  that  the  more  KM  services  brought  online, 
the  more  information  eontent  increases  and  information  quality  improves.  However, 
managing  the  information  and  knowledge  generated  seemed  to  be  a  hurdle,  as  one 
member  indieated: 


“The  amount  of  information  growth  was  phenomenal  after  the  initial 
standup  of  the  Intranet.  CoPs  were  more  difficult  to  standup  and  populate. 
As  employees  began  to  realize  there  was  a  single  source  of  reliable 
information  they  could  count  on,  they  too  began  to  contribute.  This  leads 
to  an  issue  that  was  of  concern.  Information  quality  can  be  an  issue.  Data 


Owners/Knowledge  Center  managers  were  needed  to  reduce  or  eliminate 
invalidated  or  poor  quality  information.  But  these  tend  to  be  your 
brightest  and  busiest  people.  Convincing  them  to  expend  the  effort  was 
difficult,  even  for  those  who  understood  the  long-term  benefits.  The 
human  factor  (i.e.  INCENTIVES)  are  crucial  to  obtain  support  for  KM!” 

Of  the  four  respondents  who  chose  a  rating  of  3  or  below,  only  one  chose  to  leave 
a  comment.  His  comment  again  centers  of  the  difficulty  of  managing  such  a  large  flux  of 
information: 

“This  is  a  hard  one,  since  information  content  has  exploded  by,  and  this  is 
just  a  guess,  over  fifty  percent  increase  per  year.  However,  without 
effective  knowledge  representation,  an  ubiquitous  nested  network  of  CoPs, 
or  a  systematic  approach  to  upgrading  data  and  information  to  actionable 
knowledge  and  genuine  understanding,  we  have  increased  our  ability  to 
handle  information  only  marginally  perhaps  (again,  a  guess),  by  around 
three  percent  per  annum.  The  result  is  marginal  increase  in  usable 
information  content.” 

Benefits  Concerning  Employee  Satisfaction 

These  items  centered  on  benefits  related  to  organizational  employee  satisfaction. 
Only  one  of  the  five  items  in  this  section  achieved  consensus,  while  three  items  showed  a 


movement  toward  eonsensus  after  the  second  round.  The  last  item  failed  to  achieve 
either  consensus  or  a  movement  toward  consensus. 


Item  12:  Increased  motivation 

This  item  did  not  achieve  consensus  after  the  second  round.  The  standard 
deviation  of  the  final  mean  rating  did  decrease  between  rounds,  however,  signaling  a 
move  toward  consensus.  A  final  group  mean  rating  of  3.82  was  observed,  indicating  a 
belief  that  increased  motivation  was  generally  a  benefit  of  KM  in  the  DoD.  Table  4.19 
lists  the  ratings  and  consensus  calculations  for  this  item. 


Increased  Motivation 


Round  1  Rating 

Round  2 
Rating 

Respondent  1 

6 

6 

Respondent  2 

5 

4 

Respondent  3 

5 

5 

Respondent  4 

4 

4 

Respondent  5 

3 

3 

Respondent  6 

3 

3 

Respondent  7 

5 

5 

Respondent  8 

2 

2 

Respondent  9 

1 

4 

Respondent  10 

2 

2 

Respondent  1 1 

4 

4 

Mean 

3.64 

3.82 

Std  Deviation 

1.57 

1.25 

Upper  /  Lower  Bound 

5.20-2.07 

5.07-2.57 

BENEFIT 

Table  4.19  -  Final  ratings  and  consensus  calculations  for  increased  motivation 


Seven  of  the  eleven  participants  submitted  a  rating  of  4  or  above  for  this  item.  Of 
those  seven  that  submitted  comments,  there  was  a  clear  indication  of  an  increase  in 
motivation  and  satisfaction  related  to  the  KM  program.  One  respondent  commented, 
“Internally,  our  support  staff  feels  job  satisfaction  but  as  an  organization,  I  think  that 
we’ve  made  people’s  jobs  easier  so  I  would  think  that  increases  their  motivation  to  use 
the  KM  tools  and  to  pursue  other  efforts  since  they’ve  saved  time  due  to  KM.”  Another 
stated  that,  “most  employees  who  are  involved  with  our  KM  efforts  have  been  most 
enthusiastic.” 

The  remaining  four  respondents  submitted  a  final  rating  of  3  or  below.  Of  the 
three  that  submitted  comments,  one  member  indicated  once  again  that  his  organization’s 
KM  system  was  not  mature  enough  yet  to  reap  this  benefit  yet.  Another  believed  that, 
“Those  that  took  the  time  to  use  it  really  liked  it,  but  there  are  still  those  who  feel  they  are 
too  busy  to  sit  down  and  learn  even  thought  hey  agree  it  will  improve  their  work.”  The 
final  respondent  agreed  that  his  employees  saw  the  value  of  KM,  but  could  not  say  that 
he  necessarily  observed  an  increase  in  motivation  because  of  their  KM  system. 


Item  13:  Improved  teamwork 

This  item  achieved  consensus  after  the  first  round.  The  final  group  mean  rating  was  3.91, 
indicating  a  belief  that  improved  teamwork  was  generally  a  benefit  related  to  KM  in  the 
DoD.  Table  4.20  lists  the  ratings  and  consensus  calculations  for  this  item. 

Despite  the  Delphi  methodology  indicating  consensus  on  this  item,  only  five  of 
the  eleven  members  of  the  panel  submitted  a  rating  of  4  or  above.  However,  three  of 


these  individuals  felt  strongly  enough  about  this  particular  benefit  to  submit  a  6  rating, 
while  the  two  others  submitted  a  5.  Comments  from  these  members  included: 

“Organizationally,  we’re  working  more  cross-functionally.  Internally, 
we’re  definitely  experiencing  teamwork.” 

“Great  improvement  in  the  ability  of  the  integration  team  to  manage  seven 
different  groups  developing  products.  More  cross-fertilization  and 
synchronization  of  the  outcomes.” 

“We  have  used  collaboration  very  effectively  to  improve  teamwork  and 
knowledge  sharing.” 

Three  of  the  eleven  panel  members  submitted  a  rating  of  3  for  this  item,  while  the 
remaining  three  submitted  a  two.  Comments  from  this  end  of  the  panel  indicated  the 
difficulty  in  this  area  stemmed  around  the  inclination  for  people  to  want  to  keep 
knowledge  to  themselves: 

“Mixed  bag.  The  potential  is  strong.  Early  adopters  are  already  good 
team  players  and  very  productive.  There  are  also  laggers  that  find 
knowledge  sharing  to  be  threatening  to  individual  gains.” 


“Our  applications  respect  the  needs  that  teams  have  to  keep  certain 
information  ‘close  hold’  within  the  team,  while  also  allowing  for  the 
sharing  of  that  information  to  a  broader  audience.  To  that  extent,  we  are 
supporting  teamwork  and  not  undermining  it  by  making  sharing 
mandatory.” 


Another  of  these  respondents  eommented,  “People  are  less  dependent  on  others,  so  they 
tend  to  talk  less  to  co-workers. .  .this  is  not  good.” 


Improved  Teamwork 


Round  1  Round  2 


Respondent  1 

6 

N/A 

Consensus 
aehieved  in 
first  round 

Respondent  2 

6 

Respondent  3 

6 

Respondent  4 

3 

Respondent  5 

5 

Respondent  6 

3 

Respondent  7 

3 

Respondent  8 

2 

Respondent  9 

5 

Respondent  10 

2 

Respondent  1 1 

2 

Mean 

3.91 

N/A 

Std  Deviation 

1.70 

Upper  /  Lower  Bound 

5.61-2.21 

BENEFIT 

Table  4.20  -  Final  ratings  and  consensus  caleulations  for  improved  teamwork 


Item  14:  Shorter  training  periods 


This  item  did  not  achieve  consensus  after  the  first  round.  A  decrease  in  the 
standard  deviation  was  not  observed,  making  this  the  only  item  that  also  did  not  achieve  a 
movement  toward  consensus.  Part  of  this  may  be  attributed  to  one  member  radically 
changing  his  rating  from  a  1  to  a  6.  The  final  mean  rating  was  3.27,  indicating  the  group 
believed  that  shorter  training  periods  was  generally  not  a  benefit  related  to  KM  in  the 
DoD.  Table  4.21  lists  the  ratings  and  consensus  calculations  for  this  item. 


Shorter  Training  Periods 


Round  1  Rating 

Round  2 
Rating 

Respondent  1 

6 

6 

Respondent  2 

3 

3 

Respondent  3 

4 

3 

Respondent  4 

4 

4 

Respondent  5 

2 

2 

Respondent  6 

2 

2 

Respondent  7 

2 

2 

Respondent  8 

2 

2 

Respondent  9 

1 

6 

Respondent  10 

3 

3 

Respondent  1 1 

4 

3 

Mean 

3.00 

3.27 

Std  Deviation 

1.41 

1.49 

Upper  /  Lower  Bound 

4.41  -  1.59 

4.76-1.78 

NOT  A  BENEFIT 

Table  4.21  -  Final  ratings  and  consensus  calculations  for  shorter  training  periods 


Two  of  the  eleven  respondents  felt  very  strongly  that  shorter  training  periods  was 
indeed  a  benefit,  submitting  a  rating  of  6  for  this  particular  item.  One  of  these  individuals 


actually  changed  their  round  one  rating  of  1  to  a  6  in  round  two  because  of  a 
reinterpretation  of  KM  to  inelude  distributed  learning.  He  noted,  “We  have  a  formal 
program  for  implementing  distributed  learning  to  reduee  on-site  training.” 

Another  respondent  submitted  a  4  rating  for  this  item,  stating; 

“As  part  of  our  new  personnel  orientation,  a  thorough  introduetion  and 
review  of  the  intranet  and  its  eomponents  was  provided.  New 
employees/Marines  now  had  immediate  aceess  (via  the  search  engine)  to 
data  and  SME’s  (Subjeet  Matter  Experts)  that  enabled  them  to  perform  at 
a  level  that  previously  would  have  taken  at  least  6  months  to  attain.” 

Despite  the  enthusiasm  for  these  three  members,  the  overwhelming  majority  of 
the  panel  submitted  a  rating  of  3  or  below.  Most  did  not  provide  comments,  however  one 
individual  noted: 

“To  the  extent  that  our  centralized  repositories  of  information  are  available 
for  future  referenee  (e.g.,  for  new  employees),  it  is  easier  for  folks  to  find 
what  they  need  in  order  to  get  up  to  speed  on  new  projects  to  which  they 
have  been  assigned.  However,  learning  new  jobs  requires  a  richer  set  of 
knowledge  that  is  usually  eontextually  unique.  We  have  addressed 
overarching  information  and  knowledge  needs  of  the  Agency,  not  the 
specifie  training  needs  of  individual  positions.” 


Item  15:  Development  of  job  skill 


This  was  another  item  that  did  not  aehieve  consensus  after  the  first  round. 
However,  a  move  toward  consensus  was  observed  due  to  a  decrease  in  the  standard 
deviation  of  the  mean  rating  between  round  one  and  two.  A  final  mean  rating  of  3.09 
was  observed,  indicating  the  panel  felt  that  development  of  job  skill  was  generally  not  a 
benefit  related  to  KM  in  the  DoD.  Table  4.22  lists  the  ratings  and  consensus  calculations 
for  this  item. 

Only  two  respondents  awarded  this  item  with  a  mean  rating  of  4  or  above.  One 
indicated  a  rating  of  6  but  did  not  provide  comments,  while  the  other  submitted  a  5, 
stating: 


“We’ve  definitely  made  this  [development  of  job  skill]  easier  for  the 
organization.  They  [employees]  can  manage  their  efforts  better  and  easier, 
and  integrate  results  all  at  the  touch  of  a  key.  We’re  helping  everyone 
develop  better  management/job  skills.” 

The  remaining  panel  members  submitted  a  rating  of  3  or  below,  with  five 
indicating  a  3  rating  and  four  indicating  a  2  rating.  However,  the  limited  range  of 
comments  did  not  provide  a  clear  indication  as  to  why  the  low  ratings  were  chosen.  One 
member  indicated  that,  “. .  .learning  a  new  way  of  thinking  about  developing  products  and 
staffing  documents  has  opened  people’s  minds  about  the  possibility.”  Another  stated 
simply,  “Have  not  seen  much  in  this  area.” 


Development  of  Job  Skill 


Round  1  Rating 

Round  2 
Rating 

Respondent  1 

6 

6 

Respondent  2 

5 

5 

Respondent  3 

3 

3 

Respondent  4 

3 

3 

Respondent  5 

4 

3 

Respondent  6 

2 

2 

Respondent  7 

2 

2 

Respondent  8 

2 

2 

Respondent  9 

1 

3 

Respondent  10 

2 

2 

Respondent  1 1 

4 

3 

Mean 

3.09 

3.09 

Std  Deviation 

1.51 

1.30 

Upper  /  Lower  Bound 

4.60-1.58 

4.39-1.79 

NOT  A  BENEFIT 

Table  4.22  -  Final  ratings  and  eonsensus  calculations  for  development  of  job  skill 


Item  16:  Enhancement  of  personal  knowledge 

This  was  the  final  Delphi  questionnaire  item,  and  once  again  did  not  achieve 
consensus  after  the  second  round.  However,  a  move  toward  consensus  was  observed  due 
to  a  decrease  in  the  standard  deviation  of  the  mean  rating  between  round  one  and  round 
two.  The  final  rating  of  4.27  tied  with  increased  information  content  as  the  highest  rating 
of  the  16  items,  and  indicates  that  enhancement  of  personal  knowledge  was  considered  to 
be  a  general  benefit  related  to  KM  in  the  DoD.  Table  4.23  lists  the  ratings  and  consensus 


calculations  for  this  item. 


Enhancement  of  Personal  Knowledge 


Round  I  Rating 

Round  2 
Rating 

Respondent  1 

6 

6 

Respondent  2 

6 

5 

Respondent  3 

6 

5 

Respondent  4 

6 

5 

Respondent  5 

5 

5 

Respondent  6 

2 

2 

Respondent  7 

3 

3 

Respondent  8 

2 

2 

Respondent  9 

6 

6 

Respondent  10 

2 

4 

Respondent  1 1 

4 

4 

Mean 

4.36 

4.27 

Std  Deviation 

1.80 

1.42 

Upper  /  Lower  Bound 

6.17-2.56 

5.69-2.85 

BENEFIT 

Table  4.23  -  Final  ratings  and  consensus  caleulations  for  enhancement  of 

personal  knowledge 


Eight  of  the  eleven  respondents  submitted  a  rating  of  4  or  above,  with  two 
indieating  that  they  saw  this  as  the  best  benefit  of  KM: 


“Probably  the  single  best  benefit  to  the  workforce.  Peoples’  awareness  of 
upeoming  events,  training,  references,  how-to’s,  etc.  was  at  a  level  never 
before  seen  in  the  command.  As  a  result,  seemed  to  be  better  informed 
than  ever.  In  fact,  when  attempts  were  made  to  scale  by  eertain  portions 
of  the  Intranet,  there  was  a  hue  and  ery  to  keep  all  funetions  in  place.  The 
saying  ‘you  don’t  know  what  you’ve  got  until  it’s  gone’  applies  to  useful 
information  and  KM  in  general.” 


“Probably  the  biggest  plus. .  .the  cross-functional  sharing  of  knowledge 
across  the  organization. .  .the  beginning  of  real  cultural  change  towards 
open  communication  and  knowledge  sharing.” 

Another  individual  believed  that  this  benefit  was,  “only  limited  by  the  proactive-ness  of 
the  individual.”  Still  another  agreed  to  an  extraordinary  enhancement  of  knowledge, 
however,  “the  challenge  is  to  make  knowledge  available  easily  and  quickly.” 

Of  the  three  remaining  members  who  awarded  a  3  or  below  rating,  one  indicated 
again  that  their  system  was  not  yet  at  the  level  of  providing  this  benefit,  though  it  would 
be  in  the  immediate  future.  The  other  commented  that  they  were,  “. .  .just  beginning  to 
tackle  the  hard  problem  of  sharing  tacit  knowledge,  so  I  would  not  say  that  we  have  yet 
significantly  enhanced  personal  knowledge.” 

Measuring  KM  Benefits 

The  round  two  questionnaire  featured  one  final  open-ended  question  designed  to 
get  a  general  feel  for  whether  or  not  the  individuals  had  difficulty  measuring  the  benefit 
of  KM  in  their  organizations.  Respondents  were  asked,  “Do  you  experience  problems 
measuring  KM  benefits  in  your  organization?  Please  explain.”  No  attempt  at  reaching  a 
consensus  was  made  regarding  this  item.  Rather  the  goal  was  again  to  “test  the  waters” 
and  get  the  expert’s  general  feelings  on  this  issue.  All  eleven  panel  members  indicated  a 
difficulty  measuring  the  benefits  of  KM  in  their  organization. 


Several  panel  members  commented  on  the  particular  difficulty  measuring  the 
often  intangible  but  highly  beneficial  outcomes  of  KM.  As  discussed  in  the  literature 
review,  it  is  hard  to  measure  the  value  of  something  that  primarily  exists  in  the  minds  of 
organizational  members.  One  panel  member  noted  the  difficulty  of  getting  senior 
leadership  to  break  away  from  the  traditional  model  of  organizational  performance 
measures  (i.e.  cost  and  time  savings)  and  attempt  to  recognize  some  of  the  less  tangible 
outcomes  associated  with  KM.  Another  member’s  comment  sums  the  issue  up  nicely: 

“Much  of  the  value  of  KM  is  found  in  intangibles  or  things  we  don't 
measure  well— time  saved  (unless  you  are  a  contractor  on  the  clock, 
government  time  is  considered  'free  and  paid  for');  money  saved— (most 
cost  savings  come  from  eliminating  jobs  that  used  to  do  a  task,  but  we 
generally  don't  eliminate  jobs  in  the  government).  KM  is  probably  more 
accurately  measured  in  terms  of  anecdotal  stories— it  used  to  take  me  x 
hours  to  do  this  and  I've  cut  my  time  in  half  (a  perception,  feeling,  not  a 
validated  clock  time).” 

A  few  panel  members  resorted  to  basic  measures  in  order  to  provide  at  least  some 
idea  that  their  KM  initiative  was  working.  One  of  these  methods  worked  very 
well  initially  with  systems  that  were  web-based.  As  one  panel  member  stated,  “of 
course  ‘hits’  were  the  initial  measures  of  success.  That's  easy.  But  how  do  you 
measure  improvement  in. .  .reduction  in. .  .completing  routine  tasks,  level  of 


knowledge,  etc?”  A  few  panel  members  noted  that  they  resorted  to  a  user- 
comment  form  as  a  method  of  measurment,  though  expressed  a  belief  that  this 
was  hy  no  means  an  appropriate  method. 

Another  common  theme  among  the  comments  was  the  difficulty  in  establishing  a 
baseline  in  order  to  measure  future  improvements.  According  to  one  member,  “we  did 
not  have  an  objective  performance  baseline  from  which  to  measure  improvements,  and 
we  have  not  done  a  very  good  job  of  ‘instrumenting’  our  KM-enhanced  process  to 
capture  performance-related  improvements.”  Another  member  wrote,  “main  reason  [for 
the  difficulty  measuring]  is  that  we  often  don't  know  the  total  costs  of  today's 
methods/practices,  therefore  it  is  often  difficult  to  demonstrate  improvements.” 

A  possible  major  obstacle  in  overcome  some  of  these  difficulties  the  panel 
identified  is  a  lack  of  funding. 

“We  are  having  trouble  getting  funding  for  KM,  even  though  everyone 
expresses  support  for  it,  wants  it  and  intuitively  knows  it  will  help  them. 

Too  many  other  wartime  priorities.  So  for  the  money  we  can  get,  we  need 
to  prove  its  going  to  a  good  cause.” 

A  similar  issue  was  expressed  by  multiple  panel  members.  The  problem,  however,  is  that 
accurate  performance  measures  are  needed  in  order  to  demonstrate  success  of  the  KM 
system.  This  in  turn  leads  to  appropriate  funding.  But  funding  is  needed  to  properly 
develop  a  system  of  measurement: 
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“Even  though  the  Intranet  was  a  sueeess,  it  was  hard  to  quantify  its 
success  when  it  came  time  to  defend  it  or  obtain  additional  funding.  In 
our  terminology,  it  didn't  "support  the  warfighter"  so  therefore  was 
available  for  cutback,  even  though  the  majority  of  the  workforce  relied  on 
it  on  a  daily  basis.” 

Appendix  D  contains  a  list  of  all  of  the  comments  provided  by  the  panel 
members  regarding  difficulty  measuring  KM  benefits. 

Conclusion 

The  goal  of  this  thesis  was  to  answer  three  research  questions,  which  were 
outlined  in  chapter  one: 

1)  What  does  the  literature  identify  as  the  key  benefits  of  KM 
programs  in  general? 

2)  What  do  DoD  KM  experts  identify  as  the  key  benefits  associated 
with  KM  in  the  DoD? 

3)  Do  DoD  KM  experts  experience  problems  measuring  KM 
benefits?  If  so,  how? 
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Research  Question  One 


Question  one  was  designed  to  identify  what  KM  benefits  have  already  been 
identified  in  the  literature.  The  literature  review  presented  in  Chapter  2  provided  the 
answer  to  this  question.  In  the  early  stages  of  KM  practiee,  evidence  of  benefits  were 
provided  mostly  through  observation  and  anecdotal  evidence.  However,  the  North  and 
Hornung  (2003)  study  as  well  as  the  research  done  by  KPMG  Consulting  (1999)  was 
used  to  highlight  KM  benefits  discovered  through  methodical  research  efforts.  In 
addition,  the  balanced  scorecard  approach  to  KM  benefits  developed  by  Lopez  and 
Rayboum  of  the  APQC  (2003)  was  used  to  categorize  these  benefits  and  help  develop  the 
methodology  to  answer  research  question  two  (Table  3.2). 


Research  Question  Two 

Question  two  was  central  to  this  research  effort  and  designed  to  identify  the 
particular  KM  benefits  that  apply  specifically  in  a  DoD  context.  This  question  was 
answered  through  the  Delphi  study,  which  used  proven  methodology  to  generate 
consensus  or  movement  toward  consensus  regarding  each  benefit  identified  in  research 
question  one. 

Six  benefits  presented  to  the  Delphi  experts  achieved  consensus  after  the  first 
round,  while  nine  of  the  ten  remaining  benefits  showed  a  move  toward  consensus  after 
the  second  round.  One  benefit  failed  to  achieve  either  consensus  or  a  move  toward 
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consensus,  thus  its  applicability  to  the  DoD  was  not  conclusively  determined.  Table  4.24 
combines  these  items  into  a  table  outlining  those  items  that  the  panel  agreed  were  a 
benefit  related  to  KM  in  the  DoD. 


Benefits  related  to  KM  in  the  DoD 

Mean  Rating 

Time  savings  in  doing  routine  work 

4.09 

Inereased  information  eontent 

4.27 

Improved  (employee)  teamwork 

3.91 

Aceeleration  of  proeesses 

4.00 

Re-use  of  internal  knowledge 

4.18 

Inereased  (employee)  motivation 

3.82 

Enhaneement  of  personal  knowledge 

4.27 

Table  4.24  -  Benefits  related  to  KM  in  the  DoD  identified  by  the  Delphi  panel 


Research  Question  3 

Research  question  three  was  asked  in  response  to  the  diffieulty  measuring  KM 
benefits  identified  in  the  KM  literature.  Measuring  the  benefits  of  an  initiative  is 
essential  in  order  to  determine  its  applieability  as  well  as  candidaey  for  future  efforts  and 
funding.  After  identifying  the  benefits  of  KM  from  a  DoD  perspective,  it  seemed 
important  to  get  an  idea  if  the  same  diffieulty  measuring  those  benefits  existed  in  the 
DoD  as  well. 

This  researeh  question  was  answered  through  the  open-ended  question  presented 
in  Delphi  questionnaire  number  two.  All  eleven  panel  members  polled  stated  at  least 
some  degree  of  difficulty  in  measuring  the  benefits  of  their  respeetive  organization’s  KM 
initiates.  Central  to  this  problem  was  the  diffieulty  in  defining  performance  metries  that 
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actually  measured  the  often  intangible  benefits  that  are  provided  by  KM.  Related  to  this 
issue  was  the  difficulty  in  establishing  a  baseline  for  measuring  KM  improvement, 
though  a  baseline  carmot  be  established  until  aceurate  measures  are  developed.  Finally, 
lack  of  appropriate  funding  made  it  next  to  impossible  to  even  begin  to  develop 
appropriate  measures.  Interestingly,  members  believed  this  funding  would  eome  when 
they  eould  properly  demonstrate  the  improvements  provided  via  KM.  However,  without 
proper  funding  it  is  extremely  diffieult  to  develop  the  appropriate  measures. 
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V.  CONCLUSION  AND  RECOMMENDATIONS 


Conclusions 

This  research  effort  shows  that  there  are  indeed  some  benefits  being  realized 
through  KM  practice  in  the  DoD.  While  employment  of  the  Delphi  method  worked  to 
achieve  consensus  or  movement  toward  consensus  on  many  of  the  issues,  the  comments 
provided  were  equally  important  in  providing  insight  into  how  KM  is  being  applied  in  the 
DoD.  Despite  the  fact  that  all  items  in  the  Delphi  questionnaire  were  benefits  that  were 
actually  being  realized  in  the  private  sector,  only  7  of  the  1 6  items  presented  to  the  panel 
were  seen  as  an  actual  benefit  observed  by  KM  experts  in  the  DoD. 

Of  those  seven  items,  three  were  under  the  business  process  benefits  category  and 
three  were  under  the  benefits  concerning  employee  satisfaction  category,  while  only  one 
item  under  benefits  concerning  customer  satisfaction  was  considered  a  benefit.  Perhaps 
an  explanation  for  the  imbalance  is  provided  by  North  and  Hornung’s  discovery  that  the 
benefits  derived  from  KM  are  dependent  upon  the  KM  approach  taken  (2003).  Indeed,  a 
review  of  the  descriptions  each  panel  member  provided  regarding  the  KM  programs  they 
oversaw  indicated  a  tendency  toward  technology-based  KM  solutions.  In  other  words,  it 
seems  that  the  DoD  has  at  least  initially  applied  KM  through  technology-based  initiatives 
such  as  Communities  of  Practice  and  Knowledge  Repositories.  There  wasn’t  a  clear 
indication  that  these  KM  initiatives  were  being  targeted  toward  specific  business 
objectives,  nor  were  the  organizations  trying  to  balance  the  necessary  IT  with  proper 
management  incentives.  If  the  DoD  would  like  to  see  a  more  balanced  benefit  from  the 


application  of  KM,  perhaps  it  might  consider  using  this  more  balanced  approach.  North 
and  Homung’s  study  found  that  organizations  using  such  a  “professional  KM”  approaeh 
demonstrated  robust  benefits  aeross  all  eategories. 

Still  another  explanation  for  the  laek  of  realization  of  eustomer-related  benefits 
eould  be  eulture-based.  The  DoD,  espeeially  the  branehes  of  the  Armed  Forees,  tends  to 
ignore  the  notion  that  it  has  “eustomers”  in  the  classic  sense.  After  all,  eustomers  are 
what  profit-making  businesses  eoncern  themselves  with,  and  the  DoD  is  not  a  profit¬ 
making  business.  This  idea  was  expressed  throughout  the  comments  submitted  under  the 
benefits  concerning  customer  satisfaction  section,  with  one  member  stating,  “. .  .yes,  we 
had  many  eustomers,  some  DoD  organizations  argue  they  don’t.”  Perhaps  DoD 
organizations  looking  to  apply  KM  should  not  ignore  the  benefits  sueh  initiatives  eould 
provide  in  the  area  of  eustomer  satisfaetion.  However,  these  organizations  first  must 
recognize  that  they  likely  do  have  customers,  and  then  must  identify  who  these  eustomers 
are. 

Ultimately,  the  low  seores  on  improvements  in  product  service  and  quality  as  well 
as  increase  in  customer  satisfaction  may  indieate  that  KM  in  the  DoD  is  still  very 
internally  focused.  A  fully  KM-mature  organization  finds  benefits  both  internal  as  well 
as  external  to  the  organization,  and  finds  improvements  aeross  all  business  objeetives  as 
well  as  across  organizational  boundaries  (Davenport  &  Prusak,  1998;  Ehms  &  Langen, 
2001;  Nonaka  &  Takeuchi,  1995). 

Another  reason  for  so  many  items  not  being  found  benefieial  eould  be  because 
KM  is  still  in  its  infaney  in  the  DoD.  This  is  re  fleeted  in  several  of  the  eomments  made 


by  panel  members.  These  were  mostly  along  the  lines  of,  “we  don’t  see  that  benefit  yet, 
but  we  expeet  to  onee  our  initiative  matures.”  Similarly,  respondents  made  several 
referenees  to  the  idea  that  a  eulture  of  knowledge  sharing  was  not  yet  present  in  their 
organizations.  As  the  KM  literature  shows,  building  a  eulture  to  support  KM  is  essential 
for  sueeess,  and  sueh  an  effort  takes  quite  a  bit  of  time.  The  goal  of  this  researeh  was  to 
identify  KM  benefits  that  are  eurrently  being  realized.  However,  it  is  not  safe  to 
conclude  that  items  that  were  not  found  to  be  beneficial  under  this  research  effort  will  not 
be  benefits  realized  once  DoD  KM  matures. 

Finally,  this  research  effort  reiterated  the  notion  that  KM  benefits  are  very 
difficult  to  measure,  even  in  the  DoD.  Of  course,  measuring  benefits  first  requires 
identification  of  those  benefits.  Hopefully  the  results  of  this  research  effort  might  be 
applied  toward  building  a  methodology  to  measure  the  benefits  of  KM  in  the  DoD. 

Perhaps  a  common  theme  tying  all  of  these  findings  together  is  that  KM  in  the 
DoD  is  still  in  its  very  early  stages.  Ehms  and  Lange’s  Knowledge  Management 
Maturity  Model  (2001)  can  be  used  to  assess  how  far  an  organization  has  progressed  in 
taking  advantage  of  all  that  KM  has  to  offer.  According  to  the  model  (Figure  5.1),  the 
stages  are  defined  as  follows  (Ehms  &  Langen,  2001): 

1.  Initial  -  KM  activities  are  non-systematic  and  ad-hoc.  No  language  for 
describing  organizational  phenomenon  from  a  knowledge  point  of  view. 

2.  Repeated  -  Pilot  projects  and  single  activities  labeled  as  KM. 
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3.  Defined  -  Standardized  processes  make  creation,  sharing,  and  usage  of 
knowledge  efficient. 

4.  Managed  -  Creation,  sharing,  and  usage  of  knowledge  is  organizationally 
integrated  and  improved  (measurement). 

5.  Optimizing  -  KM  is  developed  and  continuously  self-organized. 

Judging  from  the  comments  provided  by  the  panel,  it  seems  that  DoD  KM  falls 
somewhere  between  the  ‘Repeated’  and  ‘Defined’  stages.  It  is  clear  that  many 
organizations  in  the  DoD  are  starting  to  adopt  pilot  KM  projects  and  activities  (Repeated 
KM),  and  are  in  the  beginning  stages  of  trying  to  standardize  these  processes  within  the 


Figure  5.1  -  Knowledge  Management  Maturity  Model  (Ehms  &  Langen,  2001) 
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organization  (Defined  KM).  However,  the  DoD  is  still  a  long  way  from  integrating  these 
activities  across  entire  organizations,  which  is  no  doubt  hampered  by  its  lack  of  ability  to 
measure  KM  benefits  (Managed).  This  ultimately  limits  the  ability  to  obtain  increased 
funding,  which  prevents  the  spread  of  KM  throughout  the  organization. 

Limitations 

First  and  foremost,  results  of  the  Delphi  method  are  not  meant  to  be  extrapolated 
out  to  larger  population.  The  members  of  this  Delphi  panel  were  identified  as  experts 
using  criteria  developed  by  the  researcher,  and  their  responses  reflect  their  educated 
opinions  on  the  topic.  The  results  of  this  study  should  be  viewed  as  the  consensus  of  a 
group  of  experts,  and  not  scientific  fact;  they  are  entirely  dependent  on  expert  opinion, 
and  should  be  seen  as  such. 

Second,  the  answers  of  the  Delphi  panel  are  very  subjective.  While  these 
individuals  have  been  identified  as  experts,  their  opinions  can  differ  by  a  wide  margin. 
The  Delphi  method  compensates  for  this  by  using  multiple  rounds  to  achieve  consensus. 
However,  due  to  time  and  manpower  constraints,  this  study  was  limited  to  two  rounds. 
Therefore,  it  was  not  possible  to  continue  until  consensus  was  reached  on  every  item. 
While  a  movement  toward  consensus  was  observed  for  all  but  one  item  that  did  not 
achieve  consensus,  there  were  still  some  members  whose  opinion  widely  differed  from 
the  majority  of  the  group. 

In  order  to  mitigate  subjectivity,  Delphi  questions  must  be  as  clear  as  possible. 
However,  comments  related  to  the  increased  information  content  item  suggest  that  the 
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majority  of  panel  members  viewed  this  item  as  it  related  to  an  overall  organizational 
perspeetive  rather  than  the  eustomer  perspective  as  intended.  Also,  the  questiormaire 
failed  to  take  into  account  and  make  a  distinction  between  an  internal  vs.  external 
customer  perspective.  This  may  have  affected  group  response  related  to  the  customer 
benefits  category. 

Finally,  despite  the  fact  it  seems  some  DoD  organizations  are  further  ahead  than 
others  in  recognizing  and  applying  KM,  the  practice  overall  is  still  in  its  infancy.  The 
varying  opinions  of  the  Delphi  panel  members  help  illustrate  this  notion.  However, 
research  efforts  such  as  this  one  will  hopefully  assist  in  moving  toward  a  common 
blueprint  regarding  the  application  of  KM  across  the  DoD. 

Recommendations  for  Future  Research 

Now  that  this  research  effort  has  provided  a  general  idea  of  what  a  group  of 
experts  think  are  benefits  of  KM  in  the  DoD,  additional  research  should  be  done  to 
validate  this  discovery.  Follow-on  research  could  use  a  more  quantitative  methodology 
to  identify  the  benefits  of  KM.  For  example,  a  survey  of  a  large  population  of  KM 
implementers  as  well  as  users  could  be  conducted.  Ideally  such  a  survey  could  be 
extrapolated  to  the  entire  population  of  DoD  KM  users,  thus  providing  a  more 
quantifiable  measure  of  DoD  KM  benefits. 

Further  research  could  also  be  used  to  build  a  taxonomy  for  classifying  the 
benefits  of  KM  in  the  DoD  as  well  as  build  a  methodology  for  identifying  those  benefits 
needed  in  an  organization  before  building  a  KM  system.  Murray  (2002)  believes  it  is 
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absolutely  essential  to  identify  the  desired  benefits  of  the  applieation  of  KM  before 
implementation. 

Also,  a  more  definitive  list  of  the  realized  benefits  of  KM  in  the  DoD  eould  help 
develop  a  framework  for  measuring  those  benefits.  This  is  an  absolutely  essential  step  in 
order  to  both  justify  eurrent  KM  efforts  as  well  as  spark  continued  growth  and  application 
of  KM  in  the  DoD. 

Finally,  additional  research  is  necessary  to  help  develop  an  operational  definition 
of  KM  as  it  applies  to  the  DoD.  As  the  literature  review  pointed  out,  finding  a  common 
definition  for  KM  has  continued  to  elude  the  academic  community.  However,  one  of  the 
primary  difficulties  when  conducting  KM  research  is  compensating  for  the  diverse  views 
regarding  what  exactly  constitutes  a  KM  practice.  Developing  an  operational  definition 
would  not  only  solve  this  problem,  but  help  solidify  the  distinction  between  knowledge 
and  information  management  and  build  credibility  for  the  practice  as  a  whole.  This  is 
necessary  for  KM  to  progress  beyond  what  some  call  “old  wine  in  a  new  bottle”  to 
fulfilling  its  potential  as  a  key  enabler  in  the  continual  transformation  of  the  DoD. 
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APPENDIX  A  -  Delphi  Panel  Participant  List 


Name 

Position 

Organization 

Adkins,  Randy 

Chief,  Workforce 
Development  Branch 

HQ  AFMC/RDWD 

Decker,  Debbie 

KM  Collaboration 

DISA  KM 

Houck,  Dale 
(LtCol) 

Commanding  Officer 

CSSD-36,  3D  FSSG,  III  MEF 
(USMC) 

McDonald,  Doug 

Division  Chief 

DISA  CIO/COH 

McHail,  Rex 

Program  Manager,  KM 

DFA  HQ,  J-62 

Morris,  Rodler 

Knowledge  Architect 

Army  CIO  /  G-6,  Enterprise 
Integration,  Strategic  Partnering 

Schlag,  Gretchen 

KM  Team  Leader 

Dept,  of  the  Navy,  CIO 

Tefft,  Robin  (Col) 

Chief 

Army  Medical  Department 

Center  and  School,  Department 
of  Fearning  Iimovation  and 
Technology 

Winkler,  Gary 

Principal  Director, 
Enterprise  Integration 

HQDA,  US  Army  CIO  /  G-6 
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APPENDIX  B  -  Round  1  Delphi  Questionnaire 


Benefits  of  Knowledge  Management  in  the  Department  of  Defense  Questionnaire  (Part  1 

of  2) 


Instruetions: 


1 .  The  following  questionnaire  consists  of  17  short  questions  with  room  to  add 
comments.  Completion  of  this  questionnaire  should  take  no  more  than  15-20 
minutes  of  your  time. 

2.  Specific  responses  will  be  treated  anonymously.  However,  each  participant’s 
name  and  organization  will  be  included  in  a  list  of  contributors  unless  he/she 
desires  to  be  excluded.  Please  identify  below  if  you  do  not  wish  to  be  included. 

I  Do  wish  to  be  included  on  the  list  of  contributors. 

3.  When  finished,  please  save  the  completed  questionnaire  as  an  MS  Word 
document  and  e-mail  back  to  me  as  an  attachment  at  david.sasser@afit.edu. 

4.  Please  fill  out  the  “Participant  Information”  section  below  and  then  scroll  to  the 
next  page  to  begin  the  questionnaire. 


Participant  Information 

Participant  name: 

Participant  organization  /  office  symbol: 

Position  within  organization: 

Years  working  in  organization: 

Years  experience  working  in  knowledge  management: 

-  Please  review  the  list  of  knowledge  management  (KM)  benefits  below.  For  each  item, 
consider  the  degree  to  which  you  perceive  the  application  of  KM  in  your  organization 
has  provided  the  benefit  listed.  Remember  to  consider  each  benefit’s  applicability  to 
your  organization  only.  The  purpose  of  this  study  is  to  identify  realized  benefits  of 
KM.  Therefore,  do  not  respond  based  upon  how  you  think  KM  should  benefit,  rather 
respond  only  to  how  you  have  actually  witnessed  benefits  within  your  organization. 
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-  Please  rate  your  pereeived  degree  of  benefit  for  each  item  on  the  six-point  scale  shown 
below. 


Perceived  degree  of  benefit  in  my  organization 

1  2  3  4  5  6 

< - > 

NO  BENEFIT  STRONG  BENEFIT 


-For  each  item  you  perceive  as  having  a  degree  of  benefit,  please  use  the  comments 
section  to  provide  an  explanation  of  how  the  benefit  has  translated  within  your 
organization.  An  example  response  is  provided  below. 


Avoidance  of  redundancies 
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Comments:  Before  we  implemented  KM,  when  searching  out  a  technical  answer  from  our  process 
experts  we  had  to  spend  time  re-discovering  who  exactly  those  experts  were.  Now  with  the  CoP  tool 
that  we've  implemented,  personnel  can  easily  find  who  the  experts  are  under  several  pre-defined 
categories.  They  can  even  find  answers  to  some  of  the  most  frequently  asked  questions  immediately 
using  the  tool. 


-  Space  is  provided  at  the  end  of  this  questionnaire  to  list  any  benefits  you  have  observed 
that  are  not  contained  in  the  list  below. 


Recognized  benefits  of  KM 

Perceived  degree 
of  benefit  in  my 
organization 

Business  Process  Benefits 

Acceleration  of  processes 

1 

Comments: 

Avoidance  of  redundancies 

1 

Comments: 

Re-use  of  internal  knowledge 

1 

Comments: 

Reduction  of  errors 

1 

Comments: 

Time  savings  in  doing  routine  work 

1 

Comments: 
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Benefits  concerning  customer  satisfaction 

Better  response  time  1 

Comments: 


Improvements  in  product  and  service  quality  1 

Comments: 


Better  customer  communication  1 

Comments: 


Increase  in  customer  satisfaction  1 

Comments: 


Better  customer  retention  1 

Comments: 


Increased  information  content  1 

Comments: 


Benefits  concerning  employee  satisfaction 

Increased  motivation 

1 

Comments: 

Improved  teamwork 

1 

Comments: 

Shorter  training  periods 

1 

Comments: 

Development  of  job  skill 

1 

Comments: 

Enhancement  of  personal  knowledge 

1 

Comments: 


Please  use  the  space  below  to  outline  any  benefits  you  have  observed  in  your  organization  that  are  not 
on  the  list  above: 


Thank  you  for  completing  this  questionnaire.  Please  save  the  file  and  e-mail  it  as  an 
attaehment  to  david.sasser@afit.edu. 


APPENDIX  C  -  Round  2  Delphi  Questionnaire 


Benefits  of  Knowledge  Management  in  the  Department  of  Defense  Questionnaire  (Part  2 
of  2) 

Instruetions: 

1 .  Please  read  the  following  instruetions  before  filling  out  this  questioimaire.  This 
questionnaire  consists  of  three  sections.  The  first  section  contains  10  KM  benefits 
which  did  not  meet  a  rating  consensus  among  the  group.  The  second  section  is  an 
open-ended  question  regarding  the  measurement  of  KM  benefits.  The  third 
section  is  for  your  information  only  and  outlines  the  KM  benefits  that  reached  a 
rating  consensus  in  the  first  round. 

2.  Consensus  for  round  one  was  reached  by  applying  the  following  measure: 

90%  or  more  of  all  respondents’  inputs  fall  within  +/-  1  standard  deviation  (SD)  of  the 
group  mean  (SD’s  were  rounded  to  the  nearest  whole  number).  Also,  of  the  remaining 
10%,  no  more  than  one  (1)  response  had  a  conflicting  overall  opinion  than  the  group 
response  [i.e.,  all  group  responses  but  one  fell  within  the  1-3  range  (generally  no  benefit)  or 
4-6  range  (generally  a  benefit)]. 

3.  The  rating  system  for  this  questionnaire  is  the  same  as  the  rating  system  used  in 
round  one. 

Each  item  includes  the  group  mean  response,  your  round  one  response,  and  a 
place  for  you  to  record  your  round  two  response.  Use  the  drop  down  box 
provided  to  enter  your  original  or  new  rating.  Also  review  the  next  box  to 
determine  whether  your  rating  differs  from  the  general  consensus.  If  so,  please 
use  the  text  box  to  provide  an  explanation. 

4.  Each  item  also  includes  a  list  of  comments  compiled  from  all  panel  member 
responses.  Please  review  these  comments  as  well  as  the  group  mean  response 
before  deciding  whether  to  keep  or  change  your  round  one  response.  A  few 
additional  minutes  of  your  time  to  review  your  previous  selection  and  either  keep 
it  or  select  a  different  response  ensure  accurate  results  of  the  Delphi  method. 

5.  Section  2  features  an  open-ended  question  designed  to  identify  any  problems  you 
may  or  may  not  experience  measuring  KM  benefits  to  your  organization.  This 
will  provide  additional  information  for  my  data  analysis. 

6.  Section  3  is  provided  EYI  only  so  that  you  may  see  the  mean  rating  for  items  that 
achieved  consensus  as  well  as  comments  from  your  fellow  DOD  knowledge 
management  practitioners. 
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7.  Please  complete  the  questionnaire  by  14  Dec  03,  save  as  a  Word  file,  and  e-mail 
back  to  me  as  an  attachment  at  david.sasser@afit.edu. 

8.  Before  beginning,  please  take  a  quick  moment  to  fill  in  the  question  below.  A 
description  of  the  general  KM  processes  you  are  using  in  your  organization  will 
help  with  my  analysis. 


Please  provide  a  brief  description  of  the  KM  systems/programs/processes  you  are 
using  in  your  organization: 
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SECTION  1 


-  Please  review  the  following  list  of  10  knowledge  management  (KM)  benefits  below 
which  didn’t  achieve  a  rating  consensus  in  round  one.  For  each  item,  review  the  group 
mean  response  as  well  as  the  comments  provided  by  the  group  members.  Then  review 
your  original  rating  and  decided  if  you  would  like  to  keep  or  change  your  rating.  Finally, 
if  your  rating  differs  according  to  the  description  provided,  please  provide  a  brief 
explanation  why. 

-  Consider  your  rating  based  on  the  six-point  scale  shown  below,  which  is  the  same  as 
round  one. 


Perceived  degree  of  benefit  in  my  organization 


1  2 

3 

4 

5  6 

< - 

- > 

NO  BENEFIT 

STRONG  BENEFIT 

Recognized  benefits  of  KM 

Mean  group 
rating 

Your 

original 

rating 

Perceived  degree 
of  benefit  in  my 
organization 

Business  Process  Benefits 

1.  Acceleration  of  processes 

4 

1 

If  your  final  rating  is  below  4,  please  explain: 

Group  Comments: 

-  People  can  spend  time  producing  the  product  instead  of  looking  for  information 


-  We  have  several  in-house-developed  programs  for  tracking  programs  and  tasks  that  our  customers 
say  have  saved  them  days  each  month  (what  they  used  to  do  manually) 

-  With  the  implementation  of  KM  we  have  been  able  to  automate  and  streamline  several  reporting 
processes.  The  data  is  more  accuate  and  timely. 

-  In  some  cases,  the  use  of  KM  tools  merely  "sped  up  the  mess."  That  is,  the  tools  merely 
automated  a  process  that  was  not  efficient  in  the  first  place. 

-  Under  the  umbrella  of  our  KM  initiative,  we  have  undertaken  steps  to  improve  the  information 
management  of  the  agency  (good  KM  requires  sound  IM  foundation).  We  are  moving  away  from 
Powerpoint-based  methods  for  capturing  and  sharing  information  to  database-driven  approaches.  In 
the  process,  we  have  automated  processes  that  had  previously  been  manually  performed. 

-  We  are  in  the  early  phases  of  implementation,  so  adoption  of  new  practices  and  familiarity  of 
supporting  tools  is  still  an  issue. 


-  In  the  areas  of  serial  and  strategic  knowledge  transfer,  the  Army  has  done  exceptionally  well,  with 
very  effective  use  of  action  reviews  and  a  lessons  learned  program  that  worked  exceptionally  well 
before  11  September.  On  the  other  hand,  the  Army  is  only  at  the  beginning  of  using  fundamental 
KM  approaches  to  transform  most  of  its  central  business  processes. 

-  For  the  processes  that  have  successfully  used  KM  techniques,  the  payoff  is  very  high.  When  we 
consider  the  percentage  of  the  enterprise  that  have  enabled  their  business  using  KM,  the  benefit  is 
lower. 

-  People  don't  use  km  processes  enough  to  fully  take  advantage  of  them. .it's  a  culture  issue. 


2.  Re-use  of  internal  knowledge _ _ 4 _ _ _ 1 _ 

If  your  final  rating  is  below  4,  please  explain: 

Group  Comments: 

-  Gradually  people  are  getting  used  to  our  one  stop  shop  concept  for  our  KM  Portal 

-  For  those  items  we  are  capturing  and  sharing  we  are  seeing  an  increase  in  the  re-use  of 
knowledge  and  best  practices. 

-  One  of  the  more  significant  benefits  of  KM.  We  now  had  mechanisms  that  made  it  much  easier  to 
capture,  store,  access  and  most  importantly,  reuse,  information/knowledge.  The  most  difficult 
aspect  to  overcome  was  getting  people  to  contribute.  KM  does  not  seem  to  flourish  unless  tied  to 
performance  appraisals. 

-  Easier  for  other  teams  and  working  groups  within  the  teams  to  see  what  others  are  doing.  Flave 
observed  several  instances  where  groups  re-used  what  others  had  done,  where  before  they  would 
have  created  something  similar  but  slightly  different  without  a  KM  system. 

-  Now  that  we  are  beginning  to  capture  data  that  had  previously  been  located  in  filing  cabinets,  PC 
hard  drives,  etc.  and  storing  it  in  centralized  databases,  we  are  able  to  reuse  information  to  satisfy 
different  reporting  requirements. 

- 1  think  we  do  well  in  this  area.  We  are  documenting  processes,  posting  them  for  easy  access,  and 
providing  subject  matter  experts  as  contact  points. 

-  The  Army  led  the  way  in  managing  explicit  knowledge,  and  has  perhaps  the  global  best  practice  in 
tacit  to  explicit  knowledge  exchange  in  its  CompanyCommand.mil  and  its  lessons  learned  program, 
although  fusion  of  legacy  lessons  lamed  with  new  structures  like  communities  of  practice  and  virtual 
teams  is  just  underway  in  terms  of  serious  institutionalization.  Given  the  exceptional  demands  of  an 
organization  in  transformation  and  at  war,  much  needs  to  be  done  to  mature  as  a  learning 
orgnaization,  and  to  launch  as  a  teaching  organization. 

-  Sharing  lessons  learned  is  theoretically  useful,  but  no  specific  information  is  available  to  show  that 
people  use  this  to  improve  their  mission. 


Benefits  concerning  customer  satisfaction 

3.  Better  response  time  3  1 

If  your  final  rating  is  above  3,  please  explain: 

Group  Comments: 

-  One  guard  unit  deploying  overseas  indicated  the  CoP  saved  them  significant  time  in  getting  up  to 


on 


speed 

-  We  are  very  customer  focused  and  trying  to  improve  our  response  time  with  internal  process 
improvements  and  prioritization  of  work. 

-  Feedback  has  been  positive.  Customers  are  seeing  their  requirements  implemented  in  a  more 
timely  manner. 

-  Our  KM  initiative  is  focused  on  the  internal  information  and  knowledge  sharing  needs  of  the 
Agency.  Therefore,  the  benefits  do  not  accrue  to  our  customers  except  for  VERY  indirectly. 

-  Fileding  our  coming  Battle  Command  Knowledge  System  (BCKS)  will  transform  this  into  our 
greatest  strength  in  the  next  two  years. 

-  Judicious  use  of  KM  has  shown  significant  reduction  in  cycle  time  for  those  organizations  that 
understand  how  to  implement  it  effectively.  Once  again,  this  improvement  is  seen  on  a  case-by-case 
basis,  not  enterprise-wide. 

-  Customers  can  help  themselves  better.. don't  have  to  rely  as  much  on  others..they  feel  more  in 
control 


4.  Improvements  in  product  and  service 
quality 

3 

1 

If  your  final  rating  is  above  3,  please  explain: 

Group  Comments: 

-  Some  CoPs  are  better  able  to  provide  new  services  as  those  are  shared  across  the  community 


-  Nothing  existed  before  so  we  have  the  advantage  of  having  new  products  for  the  customer 

-  Bringing  visibility  to  data  has  improved  quality. 

-  Consistency  through  re-use,  with  opportunities  to  learn  from  earlier  experiences 

-  The  BCKS  will  transform  this  as  well,  but  meanwhile  our  development  of  TTPs,  doctrine  and 
training  materials,  and  materiel  acquistion  leave  much  to  be  desired. 

-  Better  information  is  obtained  to  facilitate  actions 


5.  Better  customer  communication  3  1 

If  your  final  rating  is  above  3,  please  explain: 

Group  Comments: 

-  Our  customers  are  happy  with  our  communication  and  our  desire  to  serve  them  better 

-  Almost  immediate.  Requirements  and  feedback  times  have  improved. 

-  Not  sure  the  KM  system  really  improved  the  overall  product  to  the  'customer',  however  it  did  make 
it  easier  to  develop  that  product. 
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-  Collaboration  and  deliberate  establishment  of  Communities  of  Practice  have  already  begun  to  be 
appreciated. 

-  Connetivity  between  what  the  fighting  Army  needs  and  what  the  institutional  Army  provides  is 
broken,  although  we  believe  that  BCKS  will  fix  this  too. 

-  Faster  response  time  to  questions 


6.  Increase  in  customer  satisfaction  3  1 

If  your  final  rating  is  above  3,  please  explain: 

Group  Comments: 

-  We're  still  pretty  new  so  it's  hard  to  say  but  I  don't  think  our  satisfaction  rate  is  decreasing 

-  Our  customers  are  talking  about  our  new  capabilities  and  improved  level  of  service. 

- 1  have  no  evidence  to  support  an  increase  in  customer  satisfaction  or  retention  (yes,  we  had  many 
customers,  some  DOD  organizations  argue  they  don't).  While  I  am  sure  that  employees  were  able 
to  find  needed  information  more  quickly  and  work  more  efficiently,  I  cannot  be  sure  what  correlation 
it  had  in  terms  of  customer  support.  The  benefits  of  knowledge  sharing  are  of  little  use  if  they  can't 
be  translated  the  "last  tactical  mile"  to  the  customer. 

-  Jury  is  still  out,  but  most  people  anticipate  good  improvements  in  this  area  are  nearly  here. 

-  Customers  see  potential  improvements  coming  from  entral  KM  initiatives,  and  thus  we  are  reaping 
customer  satisfaction  in  advance. 

-  Product  quality,  service  and  response  time  increases  lead  to  better  customer  satisfaction 


Benefits  concerning  employee  satisfaction 

7.  Increased  motivation  4  1 

If  your  final  rating  is  below  4,  please  explain: 

Group  Comments: 

-  This  is  hard  to  answer.  Internally,  our  support  staff  feels  job  satisfaction  but  as  an  organization,  I 
think  that  we've  made  people's  jobs  easier  so  I  would  think  that  increases  their  motivation  to  use  the 
KM  tools  and  to  pursue  other  efforts  since  they've  saved  time  due  to  KM. 

-  Employees  get  almost  immediate  feedback. 

-  My  personal  satisfaction  increased  as  more  information  became  available  to  me.  I  felt  more  "in 
tune"  with  the  direction  and  daily  events  occurring  within  the  command,  very  difficult  to  do  before 
the  advent  of  the  Intranet  and  KM  principles. 

-  Those  that  took  the  time  to  use  it  really  liked  it,  but  there  are  still  those  who  feel  they  are  too  busy 
to  sit  down  and  learn  even  though  they  agree  it  will  improve  their  work. 

-  Employees  seem  to  recognize  that  the  intiatives  that  we  have  selected  to  date  provide  value  and 
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believe  that  we  are  on  the  right  track.  To  this  extent,  our  work  is  positive.  I  cannot  say  that  it  alone 
has  made  a  significant  contribution  to  improving  motivation. 

-  Most  employees  who  are  involved  with  our  KM  efforts  have  been  most  enthusiastic.  They  see  the 
values.  They  generally  want  to  share  knowledge. 

-  We  have  not  yet  fielded  the  nested  network  of  CoPs  and  the  methods  of  the  teaching  organization, 
with  its  interactive  knowledge  creation  woven  into  the  fabric  of  daily  decision  making  and 
performance,  likely  to  push  us  past  the  tipping  point  here. 


8.  Shorter  training  periods  3  1 

If  your  final  rating  is  above  3,  please  explain: 

Group  Comments: 

-  Since  we  implemented  standards  and  common  user  interface,  we  have  seen  a  decrease  in  the 
number  of  training  requests  and  help  desk  calls. 

-  Those  that  took  the  time  to  learn  adapted  to  the  system  quickly.  Even  though  we  provided  hands- 
on  instruction  and  full-time  desk-side  assistance,  there  are  still  a  minority  who  feel  they  are  just  too 
busy  to  learn  a  better  way  of  doing  things. 

-  To  the  extent  that  our  centralized  repositories  of  information  are  available  for  future  reference 
(e.g.,  for  new  employees),  it  is  easier  for  folks  to  find  what  they  need  in  order  to  get  up  to  speed  on 
new  projects  to  which  they  have  been  assigned.  However,  learning  new  jobs  requires  a  richer  set  of 
knowledge  that  is  usually  contextually  unique.  We  have  addressed  overarching  information  and 
knowledge  needs  of  the  Agency,  not  the  specific  training  needs  of  individual  positions. 

-  The  Army  is  only  now  beginning  to  apply  contemporary  KM  to  its  training  cycle.  BCKS  is  the  key. 

-  People  get  up  to  speed  faster 


9.  Development  of  job  skill  3  1 

If  your  final  rating  is  above  3,  please  explain: 

Group  Comments: 

-  We've  definitely  made  this  easier  for  the  organization.  They  can  manage  their  efforts  better  and 
easier,  and  integrate  results  all  at  the  touch  of  a  key.  We're  helping  everyone  develop  better 
management/job  skills 

-  Have  not  seen  much  in  this  area.  We  have  created  links  to  knowledge  that  historically  has  not 
been  readily  available. 

-  learning  a  new  way  of  thinking  about  developing  products  and  staffing  documents  has  opened 
people's  mind  about  the  possibility.  Some  users  found  new  and  advanced  ways  of  employing  the 
system,  beyond  the  basic  set  of  capabailities  and  processes  that  we  anticipated. 

-  Our  initial  efforts  have  focused  more  on  information  management  than  pure  knowledge 
management.  We  are  now  just  beginning  to  tackle  the  hard  problem  of  sharing  tacit  knowledge,  so 
I  would  not  say  that  we  have  yet  significantly  enhanced  job  skills  with  our  existing  initiatives  (which 
are  mostly  dealing  with  explicit  knowledge). 

-  rudimentary  except  in  those  few  areas  with  -rate  CoPs,  like  Company  Command. 

-  more  info,  more  knowledge  is  passed  between  people 


10.  Enhancement  of  personal  knowledge  4  1 


If  your  final  rating  is  below  4,  please  explain: 

Group  Comments: 

-  Probably  the  biggest  plus.,  the  cross  functional  sharing  of  knowledge  across  the  organization.,  the 
beginning  of  real  cultural  change  towards  open  communication  and  knowledge  sharing. 

-  Our  field  sites  are  more  engaged  with  headquarters  and  more  aware  of  current  status  of  products 
and  services. 

-  Probably  the  single  best  benefit  to  the  workforce.  People's  awareness  of  upcoming  events, 
training,  references,  how-to's,  etc.  was  at  a  level  never  before  seen  in  the  command  (an  acquisition 
command).  As  a  result,  seemed  to  be  better  informed  than  ever.  In  fact,  when  attempts  were 
made  to  scale  by  certain  portions  of  the  Intranet,  there  was  a  hue  and  cry  to  keep  all  functions  in 
place.  The  saying  "you  don't  know  what  you've  got  until  it's  gone"  applies  to  useful  information  and 
KM  in  general. 

-  Our  initial  efforts  have  focused  more  on  information  management  than  pure  knowledge 
management.  We  are  now  just  beginning  to  tackle  the  hard  problem  of  sharing  tacit  knowledge,  so 
I  would  not  say  that  we  have  yet  significantly  enhanced  personal  knowledge. 

-  Those  directly  involved  with  the  KM  efforts  sre,  in  my  opinion,  deriving  lots  of  benefit  already. 
Enterprise-wide,  we  are  hopeful,  but  can't  claim  wide-spread  acceptance  yet. 

-  The  BCKS  will  revolutonize  this  area,  with  the  nested  network  of  CoPs  central  to  self-development 
and  lifelong  learning.  Right  now  self-development  is  "tooth-pick  thin." 

-  The  availability  of  relevant  information  provides  extraordinary  enhancement  of  knowledge.  The 
challenge  is  to  make  knowledge  available  easily  and  quickly. 

-  Only  limited  by  the  proactiveness  of  the  individual 
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Open  ended  question  regarding  measuring  KM  benefits. 
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SECTION  3 


This  section  provided  FYI.  These  are  the  six  items  that  achieved  a  rating  consensus  in 
round  one,  along  with  comments  from  the  group. 


Recognized  benefits  of  KM  Mean  group  rating 


1.  Avoidance  of  redundancies  3  (generaily  regarded  as  no  benefit) 

Group  Comments: 

-  People  can  spend  time  on  producing  the  product  instead  of  looking  for  information 

-  We  are  still  trying  to  integrate  most  things  so  we're  not  there  yet. 

-  We  have  made  improvements  in  this  area,  however  since  we  are  still  in  the  pilot  phase  of  our 
implementation,  some  redundancy  still  exists.  However  as  more  and  more  people  become  aware 
and  comfortable  with  the  new  tools/capabilities,  they  are  becoming  more  comfortable  with  moving 
away  from  their  manual/redundant  processes. 

-  Due  to  re-use  of  knowledge,  there  has  been  some  reduction  in  redundancies 

-  We  have  not  yet  eliminated  redundancies,  but  we  expect  to  as  part  of  our  process.  Our 
emphasis  is  on  identifying  authoritative  sources  of  data  and  using  them.  All  of  our  new 
capabilities  have  been  built  on  existing  systems  (when  appropriate).  Thus,  we  have  not  added 
new  redundancies.  Eliminating  redundancies  will  follow. 

-  We  have  a  fairly  active  KM  "Working  IPT"  that  has  begun  the  culture  change  in  this  area  I  think. 
Associations  among  WIPT  members  seems  to  have  encouraged  knowledge  exchange  and 
reduction  of  some  need  to  "invent  it  here". 

-  The  Army  Concept  Based  Requirements  System  (CBRS)  made  the  Army  a  pioneer  in  knowledge- 
based  developments,  but  the  system  withered  after  1993,  without  systematic  use  being  made  of 
newer  KM  structures  like  Communities  of  Practice.  Modular,  object-based  developments  based  on 
effective  knowledge  representation  is  only  in  its  infancy. 

-  The  potential  exists,  but  I  don't  think  KM  is  sufficiently  embedded  for  people  to  maximize  its  use. 


2.  Reduction  of  errors  3  (generally  regarded  as  no  benefit) 

Group  Comments: 

-  Menu  driven  reduces  erros  and  automatic  charts  from  data  reduces  errors 

-  Because  we  are  still  a  pilot,  not  sure  if  we  are  seeing  a  true  reduction  in  errors  at  this  time. 
However  we  have  seen  that  errors  are  identified  earlier  on  in  the  process  before  becoming  an 
issue. 

- 1  cannot  say  that  excessive  errors  were  something  that  we  found  to  be  a  problem.  Therefore, 
we  have  not  found  it  necessary  to  develop  associated  strategies  or  initiatives  to  combat  errors. 

-  As  we  re-use  internal  knowledge  more  effectively,  errors  naturally  decline. 

-  The  Army  pioneered  globally  both  After  Action  Reviews  and  strategic  organizational  learning, 
with  a  profound  reduction  in  errors  within  leader  teams,  and  by  other  leader  teams  making  the 
same  mistake  as  their  predecessors.  Slow  implementation  of  cutting-edge  KM  processes, 
structures,  tools  and  mindsets  has  hampered  our  potential  for  reducing  errors. 


3.  Time  savings  in  doing  routine  work 

4  (generaily  regarded  as  a  benefit) 

Group  Comments: 

-  Our  CoPs  tell  us  they  are  saving  8  hours  a  week  thought  the  CoP 


-  So  far  this  is  the  biggest  pius  for  KM.  Not  oniy  is  data  entered  once,  it  creates  multipie  reports 
and  charts  so  those  no  longer  have  to  be  done  manually 

-  Have  seen  a  direct  correlation  to  time  savings  and  senior  ieadership  buy-in  to  the  new  and 
improved  processes.  If  the  senior  ieadership  is  on  board,  their  staffs  wiil  utiiize  the  new  toois. 
Several  manual  processes  have  been  automated  which  have  resulted  in  time  savings. 

Information  is  able  to  be  gathered  and  assembled  more  quickly. 

-  Again,  the  precepts  of  KM  proved  significant  in  terms  of  routine  work.  A  weli-designed  Intranet 
was  crucial  to  making  information  easier  to  find  and  allowing  employees  to  complete  routine  tasks 
in  iess  time  thereby  freeing  more  time  for  them  to  focus  on  more  important  matters  (or  so  the 
theory  goes).  Routine  items  such  as  scheduling  conference  rooms  and  compieting  leave  requests 
were  placed  on  the  intranet  where  they  were  accessible  by  all  and  easily  completed.  These  are 
just  two  examples  of  items  that  saved  time.  Some  would  state  that  this  is  not  an  example  of  KM 
in  action.  However,  it's  all  relative,  even  knowing  the  owner/scheduler  of  a  conference  room  is 
can  be  important  to  the  person  tasked  with  scheduiing  its  use. 

-  Gave  the  integration  the  ability  to  put  together  their  required  daily  briefings  much  faster.  Before 
they  had  to  wait  on  each  team  to  send  an  e-mail  with  the  latest  file  attached,  there  was  often 
confusion  as  to  which  attachement  was  the  iatest  version.  Then  the  integration  team  had  to 
review  ail  the  daily  presentations  and  eiiminate  redundancies  and  try  to  synchronize  the  input. 

Now  the  KM  system  provides  version  control  and  a  way  for  others  to  review  it  and  then  post 
comments,  so  it  is  a  collaborative  review  process. 

-  Some  of  our  new  applications  have  saved  a  tremendous  amont  of  manual  labor. 

-  Not  a  lot  yet,  but  we  anticipate  big  progress  soon.  We  are  now  impiementing  what  we  cai  "DLA 
eWorkplace"  which  embodies  many  KM  capabiiities  via  an  enterprise  portal.  This  wiil  enable  much 
quicker  access  to  various  types  of  knowiedge  resources,  in  a  much  easier  way,  and  it  provides  for 
active  employee  contribution  through  collaboration,  shared  iibraries,  and  more.  Releases  of 
incremental  versions  of  DLA  eWorkplace  have  already  begun  with  small  test  communities,  and  wili 
be  made  avaiiable  enterprise-wide  by  next  month. 

-  Our  best  practices  have  shown  dramatic  time  savins.  The  prime  community  of  practice  (CoP), 
CompanyCommand.mil,  saves  as  much  as  30  percent  for  active  participants.  While  we  have  a 
state-of-the-art  plan  for  networking  the  force  with  structured  CoPs,  it  has  not  yet  been 
implemented.  Thus,  our  impact  on  time  savings  is  iimited  to  fragments  of  the  force. 

-  Automating  and  web-enabling  the  business  process  saves  significant  time.  Once  again,  it  is 
succesfui  in  portions  of  our  business,  but  has  not  matured  to  the  enterprise  level. 


4.  Better  customer  retention  2  (generally  regarded  as  no  benefit) 

Group  Comments: 

-  Hmmm,  I  never  considered  whether  or  not  we  should  retain  our  customers!  Let's  just  say  that 
more  customers  come  to  us  as  we  integrate  our  portal  services  and  we  haven't  lost  any  except  to 
normal  attrition  (retirement,  reassignments,  etc.) 

-  We  see  a  daily  increase  in  the  number  of  requests  to  participate  in  our  piiot. 

-  Our  focus  is  internal.  As  we  make  knowledge  resources  easier  to  reach  and  more  dependable 


for  DLA  employees,  they  will  continue  to  find  our  methods  and  support  tools  worth  choosing. 

-  Viewing  the  soldier  in  the  field  as  our  prime  customer,  we  have  great  issues,  due  to  strain  put  on 
them  and  their  families  by  the  current  war,  and  the  lack  of  responsiveness  relative  to  the  need. 


5.  Increased  information  content  4  (generally  regarded  as  a  benefit) 

Group  Comments: 

-  Our  KM  services  increase  every  day  and  our  information  content  increases  and  gets  better  every 
day. 

-  New  content  is  being  populated  daily.  Content  Managers  are  working  together  to  share  lessons 
learned  and  develop/enhance  guidelines. 

-  The  amount  of  information  growth  was  phenomenal  after  the  initial  standup  of  the  Intranet. 
COP'S  were  more  difficult  to  standup  and  populate.  As  employees  began  to  realized  there  was  a 
single  source  of  reliable  information  they  could  count  on,  they  too  began  to  contribute.  This  leads 
to  an  issue  that  was  of  concern.  Information  quality  can  be  an  issue.  Data  owners/Knowledge 
Center  Mangers  were  needed  to  reduce  or  eliminate  invalidated  or  poor  quality  information.  But 
these  tend  to  be  your  brightest  and  busiest  people.  Convincing  them  to  expend  the  effort  was 
difficult,  even  for  those  who  understood  the  long-term  benefits.  The  human  factor  (i.e. 
INCENTIVES)  are  crucial  to  obtain  support  for  KM! 

-  We  have  no  lack  of  content.  The  challenge  is  organizing  and  serving  it  up  in  a  meaningful  way. 

-  This  is  a  hard  one,  since  information  content  has  exploded  by,  and  this  is  just  a  guess,  over  fifty 
percent  increase  per  year.  However,  without  effective  knowledge  representation,  an  ubiquitous 
nested  network  of  CoPs,  or  a  systematic  approach  to  upgrading  data  and  information  to  actionable 
knowledge  and  genuine  understanding,  we  have  increased  our  ability  to  handle  information  only 
marginally,  perhaps  (again,  a  guess),  by  around  three  percent  per  annum.  The  result  is  marginal 
increase  in  usable  information  content.  The  BCKS  may  fix  this  to  a  consdierable  degree. 

-  We  are  in  content  overload.  If  the  question  is  about  the  quantity  of  content  available,  the 
availability  is  high.  If  we  evaluate  the  ease  of  finding  and  retrieving  content,  that  is  a  bit  more 
problematic. 

-  there  is  plenty  of  content..the  challenge  is  quickly  finding  what  you  are  looking  for 


6.  Improved  teamwork  4  (generally  regarded  as  a  benefit) 

Group  Comments: 

-  Organizationally,  we're  working  more  cross  functionally.  Internally,  we're  definitely  experiencing 
teamwork 

-  Organizational  cooperation  has  increased  since  we  implemented  our  pilot. 

-  Great  improvement  in  ability  of  integration  team  to  manage  the  7  different  groups  developing 
products,  more  cross-fertilization  and  synchronization  of  the  outcomes. 

-  Our  applications  respect  the  needs  that  teams  have  to  keep  certain  information  "close  hold" 
within  the  team,  while  also  allowing  for  the  sharing  of  that  information  to  a  broader  audience.  To 
that  extent,  we  are  supporting  team  work  and  not  undermining  it  by  making  sharing  mandatory. 

-  Mixed  bag.  The  potential  is  strong.  Earlier  adopters  are  already  good  team  players  and  very 
productive.  There  are  also  loggers  that  find  knowledge  sharing  to  be  threatening  to  individual 
gains. 


Q7 


-  Virtual  teaming  with  its  tools,  mindsets,  structures,  processes  and  high-performance  end  state  is 
just  underway,  although  the  Army  legacy  approaches  are  very  powerful  here.  Army  teamwork  is 
excellent,  KM's  impact  on  it  rudimentary  as  of  yet. 

-  We  have  used  online  collaboration  very  effectively  to  improve  teamwork  and  knowledge  sharing 

-  People  are  less  dependent  on  others,  so  they  tend  to  talk  less  to  co-workers.. this  is  not  good 


Thank  you  for  completing  this  questionnaire.  Please  save  the  file  and  e-mail  it  as  an 
attaehment  to  david.sasser@afit.edu. 


APPENDIX  D  -  Member  Comments  Regarding  Difficulty  of  Measuring  Benefits 


•  Yes.  The  traditional  expectations  for  measuring  benefits  place  less  value  on  the 
intrinsic  benefits  of  knowledge  sharing.  Most  of  the  senior  leadership  developed 
their  skills  and  style  with  a  clear  understanding  of  this  traditional  model.  Our 
challenge  is  to  make  the  translation  apparent. 

•  Yes.  We  did  not  have  an  objective  performance  baseline  from  which  to  measure 
improvements,  and  we  have  not  done  a  very  good  job  of  "instrumenting"  our  KM- 
enhanced  processes  to  capture  performance-related  improvements.  Further,  our 
portal  is  not  yet  instrumented  to  capture  specific  usage  information,  which  might 
help  us  better  understand  how  our  various  appliations  are  being  used  (or  not). 
Therefore,  it  appears  that  the  best  we  can  do  at  the  time  is  use  the  measure  of 
"user  satisfaction"  as  a  surrogate  for  information  system  success,  and  we  are 
deploying  surveys  to  collect  benefit  information  from  a  personal  perpective. 

Based  on  the  results  of  your  research,  we  may  tailor  our  survey  to  address  benefits 
areas  that  we  may  have  overlooked. 

•  Our  organization  is  just  getting  started  in  this  area.  Previously,  Km  initaitvies 
sprouted  all  over  the  place— many  poorly  planned,  on  non-standard  tools,  and 
without  real  thought  as  to  what  the  organization  really  expected  from  it.  The 
problem  is— and  I  suspect  many  government,  or  at  least  DoD,  organizations  are  in 
the  same  boat— we  don't  know  what  the  baseline  is,  so  it's  hard  to  quantify 
improvement/benefit.  Much  of  the  value  of  KM  is  found  in  intangibles  or  things 
we  don't  measure  well— time  saved  (unless  you  are  a  contractor  on  the  clock, 
government  time  is  considered  'free  and  paid  for');  money  saved— (most  cost 
savings  come  from  eliminating  jobs  that  used  to  do  a  task,  but  we  generally  don't 
eliminate  jobs  in  the  government).  KM  is  probably  more  accurately  measured  in 
terms  of  anecdotal  stories— it  used  to  take  me  x  hours  to  do  this  and  I've  cut  my 
time  in  half  (a  perception,  feeling,  not  a  validated  clock  time).  People  feel  more 
empowered  when  they  know  what  else  is  going  on  in  their  area,  but  hard  to 
quantify  that  feeling.  We  are  just  now  incorporating  a  Return  on  Value  basis  for 
new  KM  projects.  Folks  will  have  to  establish  what  type  of  improvements  they 
think  the  system  will  give  them— quantifying  it  if  they  can  (such  as  reduce  access 
time  to  necessary  documents  by  50%)  and  then  follow  up  with  quantifiable  or 
anecdotal  evidence  that  indicates  if  the  initiative  really  hit  its  mark.  We  are  having 
trouble  getting  funding  for  KM,  even  though  everyone  expresses  support  for  it, 
wants  it  and  intuitively  knows  it  will  help  them.  Too  many  other  wartime 
priorities.  So  for  the  money  we  can  get,  we  need  to  prove  its  going  to  a  good 
cause.  I'm  convinced  if  we  can  have  some  performance  measures  that  show 
benefit  to  the  organization  we  will  have  a  stronger  case  to  compete  in  the  funding 
battle. 
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•  Absolutely.  Most  enterprises  operate  within  a  eultural  framework  that 
eneourages  and  rewards  competition  &  individual  achievement.  KM  demands  a 
shift  to  a  culture  where  collaboration,  knowledge  sharing,  and  achievement  are 
valued  with  competition  and  individual  achievement.  By  nature,  people  will  not 
share  and  that  includes  knowledge.  Knowledge,  no  matter  how  trivial,  is 
something  to  be  hoarded  as  a  sign  of  strength.  To  share  is  often  seen  as 
enhancing  the  posture  of  another  with  no  perceived  benefit.  This  was  by  far  the 
biggest  obstacle  to  overcome.  Developing  performance  measures  was  just  as 
difficult.  Even  though  the  Intranet  was  a  success,  it  was  hard  to  quantify  its 
success  when  it  came  time  to  defend  it  or  obtain  additional  funding.  In  our 
terminology,  it  didn't  "support  the  warfighter"  so  therefore  was  available  for 
cutback,  even  though  the  majority  of  the  workforce  relied  on  it  on  a  daily  basis. 
The  difficult  came  in  defining  performance  measures.  Of  course  "hits"  were  the 
initial  measures  of  success.  That's  easy,  but  how  to  you  measure  improvement  in 
,  reduction  in  completing  routine  tasks,  level  of  knowledge,  etc?  In  the  end  (by 
the  time  I  departed  as  CIO),  we  never  did  develop  quality  performance  measures. 

•  Somewhat.  Main  reason  is  that  we  often  don't  know  the  total  costs  of  today's 
methods/practices,  therefore  is  often  difficult  to  demonstrate  improvements.  We 
are  able  to  capture  detail  metrics  for  users  who  access  our  internal  tools;  however 
it  is  often  hard  to  measure  the  value  of  a  particular  piece  of 
information/knowledge.  To  date,  we  have  not  been  able  to  develop  good  tacit 
knowledge  measures. 

•  We're  still  in  our  infancy  on  this.  We're  getting  ready  to  implement  a  KM  Awards 
program  where  we  measure  certain  usage  and  reward  people  for  their  use, 
innovation,  and  knowledge  sharing.  Too  early  to  tell  on  this  and  other 
measurements.  Currently  we  track  usage  in  various  forms  (site  entry,  content 
update,  etc.) 

•  Measuring  specific  benefits  requires  resources  to  baseline  the  current  process  and 
then  implement  the  new  process  and  measure  the  changes.  It  requires  dedicated 
resources  to  accomplish  this  type  of  analysis  that  we  just  do  not  have.  However, 
we  do  survey  our  users  on  a  regular  basis  on  the  benefits. 

•  Yes.  We  have  an  Army  Knowledge  Implementation  Plan  with  specified  tasks  and 
actions  for  Army  organizations  but  are  just  now  starting  to  define  metrics,  collect 
them,  and  analyze  them.  Most  difficult  of  all  is  to  measure  the  effectiveness  of 
knowledge  management  tools  and  processes  for  the  war-fighter  or  war-fighting 
mission.  Our  goal  is  to  develop  agile  and  adaptive  soldiers,  leaders,  and  units,  but 
how  to  measure  that  adaptability  and  agility  is  still  being  wrestled  with. 
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Yes.  The  benefits  are  indirect,  and  therfore  difficult  to  measure.  There  are  many 
intervening  variables  that  contribute  to  the  success  of  a  project/mission  that  it  is 
difficult  to  determine  the  contribution  of  KM  to  the  success.  Currently  our 
measurements  are  process  such  as  frequency  of  access.  We  have  instituted  an 
evaluation  of  each  knowledge  product  by  the  users,  but  have  not  measured  the 
impact  of  this. 

Definitely  hard  to  measure  KM  benefits  and  working  in  the  DoD  management  is 
interested  in  the  ROT  We  are  currently  trying  to  define  metrics  that  will  help 
answer  those  questions.  Working  in  the  combined  area  of  BPR  and  KM  the 
metrics  are  going  to  be  easier  to  identify. 

Yes,  but  more  as  a  function  of  resources  than  a  lack  of  a  concept  of  how  to  do  it. 
Measurement  must  be  part  of  a  funded,  well-thought  out  management  and 
measurement  plan  and  draw  on  the  behavioral  and  cognitive  sciences,  and 
knowledge  economics  and  audits.  We  have  a  good  general  concept  for  executing 
ongoing  measurement  of  return  on  value  and  investment.  But,  the  funding  is 
working  rather  than  assured. 


APPENDIX  E  -  Delphi  Analysis  Spreadsheet 
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